Wearable image pickup apparatus, portable device and calibrator that communicate with image pickup apparatus, control methods therefor, and storage media storing control programs therefor

ABSTRACT

A wearable image pickup apparatus that eliminates manual change of an image pickup direction during picking up an image and can easily obtain an image that records experience while focusing attention on the experience. An image pickup apparatus includes an observation direction detection unit that is worn on a body other than a head of a user, an image pickup unit that is worn on the body of the user, a memory device that stores a set of instructions, and at least one processor that executes the set of instructions to: detect an observation direction of the user by the observation direction detection unit, pick up an image by the image pickup unit, and output an output image corresponding to the observation direction based on the image picked up by the image pickup unit.

BACKGROUND Field of the Disclosure

The present disclosure relates to a wearable image pickup apparatus, aportable device and calibrator that communicate with the image pickupapparatus, control methods therefor, and storage media storing controlprograms therefor, and in particular, relates to an image pickupapparatus used as an action camera, a portable device, a calibrator,control methods therefor, and storage media storing control programstherefor.

Description of the Related Art

When a user picks up an image of an object with a camera, the user needsto continue directing the camera toward the object. Accordingly, theuser is difficult to manage actions other than an image pickup actionbecause the user is busy in an image pickup operation. And the user isdifficult to focus attention on experience at a place of the userbecause the user focuses attention on the image pickup operation.

For example, about the image pickup operation, a parent as the usercannot play with a child as an object during an image pickup operation,and the image pickup operation becomes impossible while playing with thechild.

Moreover, about the focusing of attention, when the user picks up animage while watching a sport game, the user cannot cheer or cannotremember game contents, and the image pickup operation becomesimpossible while focusing attention to watch the sport game. Similarly,when a user picks up images during group travel, the user cannotexperience impression of the same level as other members, and whenpriority is given to experience, the image pickup is neglected.

Japanese Laid-Open Patent Publication (Kokai) No. 2007-74033 (JP2007-74033A) discloses a technique that uses a second camera that picksup a user in addition to a first camera that picks up an object. Thistechnique calculates a moving direction and visual-line direction of auser from an image picked up by the second camera, determines an imagepickup direction of the first camera, and picks up an image of an objectestimated on the basis of user's taste and state.

Moreover, Japanese Laid-Open Patent Publication (Kokai) No. 2017-60078(JP 2017-60078A) (Counterpart of US Patent Application 20170085841)disclose an image recording system including a sensor device that isattached to a user's head and an image pickup apparatus that isseparately attached to a user's body or a bag. The sensor deviceconsists of a gyro sensor or an acceleration sensor and detects a user'sobservation direction. The image pickup apparatus picks up an image inthe observation direction detected by the sensor device.

However, since the second camera of JP 2007-74033A picks up an image ofthe user from a position distant from the user, the second camera needshigh optical performance in order to calculate the moving direction andvisual-line direction of the user from the image picked up by the secondcamera. Moreover, since high arithmetic processing capability is neededfor processing the image picked up by the second camera, a scale of anapparatus becomes large. Furthermore, even if the high opticalperformance and the high arithmetic processing capability are satisfied,the user's observation direction cannot be precisely calculated.Accordingly, since an object that the user wants to pick up cannot beestimated with sufficient accuracy on the basis of the user's taste andstate, an image other than what is wanted by the user may be picked up.

Moreover, since the sensor device of JP 2017-60078A directly detects auser's observation direction, the user needs to equip the head with thesensor device, which cannot solve troublesomeness in attaching anydevice to the head as mentioned above. Moreover, when the sensor deviceconsists of a gyro sensor or an acceleration sensor, certain accuracycan be obtained in detection of a relative observation direction.However, since accuracy of detection of an absolute observationdirection, especially in the horizontal rotation direction, cannot beobtained, there is an issue in a practical application.

SUMMARY

Embodiments of the present disclosure provide a wearable image pickupapparatus, a portable device and a calibrator that communicate with theimage pickup apparatus, control methods therefor, and storage mediastoring control programs therefor, which eliminate manual change of animage pickup direction during picking up an image and can easily obtainan image that records experience while focusing attention on theexperience.

Accordingly, embodiments of the present disclosure provide an imagepickup apparatus including an observation direction detection unit thatis worn on a body other than a head of a user, an image pickup unit thatis worn on the body of the user, a memory device that stores a set ofinstructions, and at least one processor that executes the set ofinstructions to: detect an observation direction of the user by theobservation direction detection unit, pick up an image by the imagepickup unit, and output an output image corresponding to the observationdirection based on the image picked up by the image pickup unit.

According to embodiments of the present disclosure, manual change of animage pickup direction during picking up an image becomes unnecessary,and an image that records experience can be easily obtained whilefocusing attention on the experience.

Further features of the present disclosure will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is an external view showing a camera body including animage-pickup/detection unit as an image pickup apparatus according to afirst embodiment.

FIG. 1B is a view showing a state where a user wears the camera body.

FIG. 1C is a view showing a battery unit in the camera body viewed froma rear side in FIG. 1A.

FIG. 1D is an external view showing a display apparatus as a portabledevice according to the first embodiment that is separated from thecamera body.

FIG. 2A is a front view showing the image-pickup/detection unit in thecamera body.

FIG. 2B is a view showing a shape of a band part of a connection memberin the camera body.

FIG. 2C is a rear view showing the image-pickup/detection unit.

FIG. 2D is a top view showing the image-pickup/detection unit.

FIG. 2E is a view showing a configuration of a face direction detectionunit arranged inside the image-pickup/detection unit and under a facedirection detection window in the camera body.

FIG. 2F is a view showing a state where a user wears the camera bodyviewed from a left side of the user.

FIG. 3A, FIG. 3B, and FIG. 3C are views showing details of the batteryunit.

FIG. 4 is a functional block diagram showing the camera body accordingthe first embodiment.

FIG. 5 is a block diagram showing a hardware configuration of the camerabody according to the first embodiment.

FIG. 6 is a block diagram showing a hardware configuration of thedisplay apparatus according to the first embodiment.

FIG. 7A is a flowchart schematically showing an image pickup/recordingprocess according to the first embodiment executed by the camera bodyand display apparatus.

FIG. 7B is a flowchart showing a subroutine of a preparation process ina step S100 in FIG. 7A according to the first embodiment.

FIG. 7C is a flowchart showing a subroutine of a face directiondetection process in a step S200 in FIG. 7A according to the firstembodiment.

FIG. 7D is a flowchart showing a subroutine of arecording-direction/area determination process in a step S300 in FIG. 7Aaccording to the first embodiment.

FIG. 7E is a flowchart showing a subroutine of a recording-areadevelopment process in a step S500 in FIG. 7A according to the firstembodiment.

FIG. 7F is a view for describing a process in the steps S200 throughS600 in FIG. 7A in a video image mode.

FIG. 8A is a view showing an image of a user viewed from the facedirection detection window.

FIG. 8B is a view showing a case where fluorescent lamps in a roomappear as background in the image of the user viewed from the facedirection detection window.

FIG. 8C is a view showing an image obtained by imaging the user andfluorescent lamps as background shown in FIG. 8B onto a sensor of theinfrared detection device through the face direction detection window ina state where infrared LEDs of the infrared detection device are notlightened.

FIG. 8D is a view showing an image obtained by imaging the user andfluorescent lamps as background shown in FIG. 8B onto the sensor of theinfrared detection device through the face direction detection window ina state where the infrared LEDs are lightened.

FIG. 8E is a view showing a difference image that is calculated bysubtracting the image in FIG. 8C from the image in FIG. 8D.

FIG. 8F is a view showing a result obtained by adjusting shades of thedifference image in FIG. 8E so as to fit with a scale of lightintensities of reflected components of infrared light projected to aface and neck of the user.

FIG. 8G is a view obtained by superimposing reference numerals denotingparts of a user's body, a double circle showing a throat position, and ablack circle showing a chin position on FIG. 8F.

FIG. 8H is a view showing a difference image calculated by the similarmethod as FIG. 8E in directing the user's face to the right.

FIG. 8I is a view showing a result obtained by adjusting shades of thedifference image in FIG. 8H so as to fit with a scale of lightintensities of reflected components of infrared light projected to aface and neck of the user and by superimposing the double circle showingthe throat position and the black circle showing the chin position.

FIG. 8J is a view showing an image of the user who directs the faceupward by 33° viewed from the face direction detection window.

FIG. 8K is a view showing a result obtained by adjusting shades of adifference image, which is calculated by the similar method as FIG. 8Ein a case that the user directs the face upward by 33°, so as to fitwith a scale of light intensities of reflected components of infraredlight projected to a face and neck of the user and by superimposing thedouble circle showing the throat position and the black circle showingthe chin position.

FIG. 9 is a timing chart showing a lighting timing of the infrared LEDsand related signals.

FIG. 10A through FIG. 10D are views describing movements of the user'sface in a vertical direction.

FIG. 11A is a view showing a target visual field set in asuperwide-angle image picked up by an image pickup unit of the camerabody in a case where the user faces the front.

FIG. 11B is a view showing an image in the target visual field extractedfrom the superwide-angle image in FIG. 11A.

FIG. 11C is a view showing the target visual field set in thesuperwide-angle image in a case where the user is observing an A-object.

FIG. 11D is a view showing an image that is obtained by correctingdistortion and blur of an image in the target visual field in FIG. 11Cextracted from the superwide-angle image.

FIG. 11E is a view showing a target visual field set in thesuperwide-angle image in a case where the user is observing the A-objectat a field-angle set value smaller than that in FIG. 11C.

FIG. 11F is a view showing an image that is obtained by correctingdistortion and blur of an image in the target visual field in FIG. 11Eextracted from the superwide-angle image.

FIG. 12A is a view showing an example of the target visual field set inthe superwide-angle image.

FIG. 12B is a view showing an example of the target visual field set inthe superwide-angle image in a case where the field-angle set value isidentical to that of the target visual field in FIG. 12A and where theobservation direction differs.

FIG. 12C is a view showing another example of the target visual fieldset in the superwide-angle image in a case where the field-angle setvalue is identical to that of the target visual field in FIG. 12A andwhere the observation direction differs.

FIG. 12D is a view showing an example of the target visual field set inthe superwide-angle image in a case where the observation direction isidentical to that of the target visual field in FIG. 12C and where thefield-angle set value is smaller.

FIG. 12E is a view showing an example that gives an image stabilizationmargin corresponding to a predetermined image stabilization level aroundthe target visual field shown in FIG. 12A.

FIG. 12F is a view showing an example that gives an image stabilizationmargin corresponding to the same image stabilization level of the imagestabilization margin in FIG. 12E around the target visual field shown inFIG. 12B.

FIG. 12G is a view showing an example that gives an image stabilizationmargin corresponding to the same image stabilization level of the imagestabilization margin in FIG. 12E around the target visual field shown inFIG. 12D.

FIG. 13 is a view showing a menu screen for setting various set valuesof a video image mode that is displayed on a display unit of the displayapparatus before an image pickup operation of the camera body.

FIG. 14 is a flowchart showing a subroutine of a primary recordingprocess in a step S600 in FIG. 7A.

FIG. 15 is a view showing a data structure of an image file generated bythe primary recording process.

FIG. 16 is a flowchart of the subroutine of a transmission process tothe display apparatus in a step S700 in FIG. 7A.

FIG. 17 is a flowchart showing a subroutine of an optical correctionprocess in a step S800 in FIG. 7A.

FIG. 18A through FIG. 18F are views for describing a process of applyingdistortion correction in a step S803 in FIG. 17.

FIG. 19 is a flowchart showing a subroutine of an image stabilizationprocess in a step S900 in FIG. 7A.

FIG. 20A and FIG. 20B are the views showing details of a calibrator usedfor a calibration process according to a second embodiment.

FIG. 21 is a flowchart showing the calibration process according to thesecond embodiment executed by the camera body and the calibrator.

FIG. 22A is a view showing a screen displayed on a display unit of thecalibrator in a step S3103 in FIG. 21 during a calibration operation fora front direction of the user.

FIG. 22B is a view showing a state where the user holds the calibratorin the front according to an instruction shown as an instruction displayin FIG. 22A.

FIG. 22C is a schematic view showing an entire superwide-angle imagethat is caught by an image pickup lens in the state in FIG. 22B.

FIG. 22D is a schematic view showing an image that is obtained bycorrecting aberrations of the superwide-angle image shown in FIG. 22C.

FIG. 22E is a schematic view showing a face direction image that isobtained by a face direction detection unit in a step S3108 in FIG. 21during the calibration operation for the front direction of the user.

FIG. 22F is a schematic view showing an in-camera image displayed in astep S3107 in FIG. 21.

FIG. 23A is a view showing a screen displayed on the display unit of thecalibrator in the step S3103 in FIG. 21 during the calibration operationin an upper right direction of the user.

FIG. 23B is a view showing a state where the user holds the calibratorto upper right according to an instruction shown as the instructiondisplay in FIG. 23A.

FIG. 23C is a schematic view showing the entire superwide-angle imagethat is caught by the image pickup lens in the state in FIG. 23B.

FIG. 23D is a schematic view showing an image that is obtained bycorrecting aberrations of the superwide-angle image shown in FIG. 23C.

FIG. 23E is a schematic view showing a face direction image that isobtained by the face direction detection unit in the step S3108 in FIG.21 during the calibration operation for the upper right direction of theuser.

FIG. 24A, FIG. 24B, and FIG. 24C are views for describing delayextraction of an image in a third embodiment.

FIG. 25A and FIG. 25B are views showing—loci of face movements held inthe third embodiment.

FIG. 26 is a flowchart showing a visually-induced-motion-sicknessprevention process according to the third embodiment.

FIG. 27A through FIG. 27F are graphs for describing an extraction-areacorrection process according to a fourth embodiment.

FIG. 28A is a flowchart showing a recording-direction/area determinationprocess according to the fourth embodiment. FIG. 28B is a flowchartshowing the extraction-area correction process in a step 400 in FIG.28A.

FIG. 29A and FIG. 29B are schematic views for describing a relationshipbetween a user's visual field and a target visual field in a case wherea short distance object is an observation target in the firstembodiment.

FIG. 30 is an external view showing a camera body including an imagepickup apparatus according to a fifth embodiment.

FIG. 31 is a block diagram showing a hardware configuration of thecamera body according to the first embodiment.

FIG. 32A and FIG. 32B are schematic views for describing a relationshipbetween a user, a calibrator, and a target visual field during acalibration process including a parallax correction mode process in thefifth embodiment.

FIG. 33A is a flowchart showing the parallax correction mode processthat is a part of the preparation process in the step S100 in FIG. 7Aaccording to the fifth embodiment.

FIG. 33B is a flowchart showing a subroutine of arecording-direction/area determination process in the step S300 in FIG.7A according to the fifth embodiment.

FIG. 34A, FIG. 34B, and FIG. 34C are schematic views showing arelationship between a defocus map generated in a step S5302 in FIG. 33Band a recording direction.

FIG. 35 is a flowchart showing an observation direction determinationprocess according to a sixth embodiment.

FIG. 36A is a view showing relationships between an observationdirection detection state of the user and a pickup image for respectiveframes according to the sixth embodiment.

FIG. 36B is a view showing relationships between an observationdirection detection state of the user and a pickup image for respectiveframes in an object lost mode according to the sixth embodiment.

FIG. 37A, FIG. 37B, and FIG. 37C are views for describing relationshipsbetween an observation direction and a face area that can be used fordetection of the observation direction according to a seventhembodiment.

FIG. 38 is a flowchart showing an observation direction determinationprocess according to the seventh embodiment in obtaining the facedirection that is executed instead of the process in a step S6004 inFIG. 35.

FIG. 39 is a view showing a relationship between the face direction anda face direction reliability in the seventh embodiment.

FIG. 40 is a schematic view showing an observation directiondetermination process in obtaining the face direction in the seventhembodiment.

FIG. 41A, FIG. 41B, and FIG. 41C are enlarged side views showing theimage-pickup/detection unit.

FIG. 42A, FIG. 42B, and FIG. 42C are side views showing a state wherethe user wears the camera body.

FIG. 43A, FIG. 43B, and FIG. 43C are enlarged side views showing theimage-pickup/detection unit without showing the connection members.

FIG. 44A, FIG. 44B, and FIG. 44C are side views showing a state wherethe user wears the camera body without showing the connection members.

FIG. 45A through FIG. 45G are views showing various combinations of theband part and a connection surface that is a section of an electriccable united with the band part.

FIG. 46A is a block diagram showing a hardware configuration of adisplay apparatus connected to a camera body including an image pickupapparatus according to a ninth embodiment.

FIG. 46B is a functional block diagram showing the camera body accordingthe ninth embodiment.

FIG. 47 is a functional block diagram showing the camera body anddisplay apparatus according a tenth embodiment.

FIG. 48 is a flowchart schematically showing an image-pickup/recordingprocess according to the tenth embodiment executed by the camera bodyand display apparatus.

FIG. 49 is a functional block diagram showing a camera body and displayapparatus according an eleventh embodiment.

FIG. 50 is a flowchart schematically showing an image-pickup/recordingprocess according to the eleventh embodiment executed by the camera bodyand display apparatus.

FIG. 51A is an external view showing a camera body according to atwelfth embodiment 12.

FIG. 51B is a perspective view showing details of animage-pickup/detection unit that is a part of the camera body accordingto the twelfth embodiment.

FIG. 51C is a perspective view showing a state where an image pickupunit of the image-pickup/detection unit in FIG. 51B turns leftward by30°.

FIG. 51D is a perspective view showing a state where the image pickupunit is directed downward by 30°.

FIG. 52 is a functional block diagram showing the camera body accordingthe twelfth embodiment.

FIG. 53 is a block diagram showing a hardware configuration of thecamera body according to the twelfth embodiment.

FIG. 54 is a flowchart schematically showing an image-pickup/recordingprocess according to the twelfth embodiment executed by the camera bodyand display apparatus.

FIG. 55 is a flowchart showing a subroutine of a face directiondetection process in a step S12300 in FIG. 54 according to the twelfthembodiment.

FIG. 56 is a flowchart showing a subroutine of a development process ina step S12500 in FIG. 54 according to the twelfth embodiment.

FIG. 57 is a block diagram showing a hardware configuration of a camerabody according to a thirteenth embodiment.

FIG. 58A, FIG. 58B, and FIG. 58C are schematic views showing examples oflearning images used in the thirteenth embodiment.

FIG. 59 is a flowchart showing a face direction detection process usingmachine learning according to the thirteenth embodiment.

FIG. 60 is a block diagram showing a hardware configuration of a camerabody according to a fourteenth embodiment.

FIG. 61A is a schematic view showing a distance image generated by a ToF(Time of Flight) device of the camera body according to the fourteenthembodiment in a state where the ToF device is arranged in a user'sclavicle position and measures upwardly.

FIG. 61B is a schematic view showing an image that extracted a face partby applying the threshold process to the distance image in FIG. 61A.

FIG. 61C is a schematic view showing an image obtained by dividing theimage in FIG. 61B into areas according to distance information.

FIG. 61D is a schematic view showing an image that shows a throatposition and chin position extracted from the image in FIG. 61C.

FIG. 62 is a flowchart showing a face-direction calculation processaccording to the fourteenth embodiment.

FIG. 63A and FIG. 63B are views showing a configuration example of acamera fixed to a head using a conventional fixing-to-head accessory.

FIG. 64 is a view showing a configuration example of a conventionalentire-celestial-sphere camera.

FIG. 65A, FIG. 65B, and FIG. 65C are views showing examples ofconversion processes of the image picked up by theentire-celestial-sphere camera in FIG. 64.

DESCRIPTION OF THE EMBODIMENTS

Hereafter, embodiments according to the present disclosure will bedescribed in detail by referring to the drawings.

First, some methods for achieving both an image pickup operation andfocusing of attention on experience will be considered. There is amethod that fixes a camera to a head using a fixing-to-head accessory topick up an image in an observing direction. This enables the user topick up an image without being occupied with the image pickup operation.Moreover, there is also a method that picks up an image in a wide areawith an entire-celestial-sphere camera during experience. This enables auser to focus attention on the experience. After the experience, theuser extracts a necessary image part from picked-upentire-celestial-sphere image and edit it to leave the image of theexperience.

However, the former method needs a troublesome action that equips thehead with the fixing-to-head accessory 902 to which a main body of anaction camera 901 is fixed as shown in FIG. 63A. Moreover, as shown inFIG. 63B, when the user equips the head with the action camera 901 withthe fixing-to-head accessory 902, appearance is bad and also a hairstyleof the user is also disheveled. Furthermore, the user feels uneasy aboutthe existence of the fixing-to-head accessory 902 and the action camera901 because of their weights and worries about bad appearance to thirdpersons. Accordingly, the user may be difficult to pick up an imagebecause the user cannot focus attention on experience in the state shownin FIG. 63B or because the user feels resistance to the style shown inFIG. 63B.

In the meantime, the latter method needs series of operations, such asimage conversion and extraction position designation. For example, anentire-celestial-sphere camera 903 equipped with a lens 904 and an imagepickup button 905 as shown in FIG. 64 is known. The lens 904 is one of apair of fish-eye lenses for picking up half-celestial-sphere imagesprovided in both sides of a housing of the entire celestial spherecamera 903. The entire-celestial-sphere camera 903 picks up anentire-celestial-sphere image using these fish-eye lenses. Then, theentire celestial sphere image is obtained by combining the images pickedup using the pair of fish-eye lenses.

FIG. 65A, FIG. 65B, and FIG. 65C are views showing examples ofconversion processes of the image picked up by theentire-celestial-sphere camera 903.

FIG. 65A shows an example of the entire-celestial-sphere image obtainedby the entire celestial sphere camera 903, and a user 906, a child 907,and a tree 908 are included as objects. Since this image is anentire-celestial-sphere image obtained by combining projection images ofthe pair of fish-eye lenses, the user 906 is distorted greatly.Moreover, since a body part of the child 907 who is the object that theuser 906 wants to pick up is located in a peripheral part of a pickuparea of the lens 904, the body part distorts greatly in right and leftand is extended. In the meantime, since the tree 908 is the objectlocated in front of the lens 904, the tree 908 is picked up withoutgreat distortion.

In order to generate an image of a visual field at which people areusually looking from the image shown in FIG. 65A, it is necessary toextract a part of the image, to perform plane conversion, and todisplay.

FIG. 65B is an image located in front of the lens 904 that is extractedfrom the image shown in FIG. 65A. In the image in FIG. 65B, the tree 908is shown in the center in the visual field at which people are usuallylooking. However, since the image in FIG. 65B does not include the child907 who the user 906 wants to pick up, the user has to change anextraction position. Specifically, in this case, it is necessary to movethe extraction position leftward and downward by 30° from the tree 908in FIG. 65A. FIG. 65C shows a displayed image that is obtained byextracting the moved position and by performing the plane conversion. Inthis way, in order to obtain the image in FIG. 65C that the user wantsto pick up from the image in FIG. 65A, the user has to extract anecessary area and has to perform the plane conversion. Accordingly,although the user can focus attention on experience during theexperience (during image pickup), subsequent workload becomes huge.Accordingly, configurations of embodiments that solve theabove-mentioned issues are devised.

FIG. 1A through FIG. 1D are views for describing a camera systemconsisting of a camera body 1 and a display apparatus 800 that isseparated from the camera body 1. The camera body 1 includes animage-pickup/detection unit 10 as a wearable image pickup apparatusaccording to a first embodiment. Although the camera body 1 and thedisplay apparatus 800 are separated devices in this embodiment, they maybe integrated.

FIG. 1A is an external view showing the camera body 1. The camera body 1is provided with the image-pickup/detection unit 10, a battery unit(power source unit) 90, a right connection member 80R, and a leftconnection member 80L as shown in FIG. 1A. The right connection member80R connects the image-pickup/detection unit 10 and the battery unit 90on the right side of a user's body (left side in FIG. 1A). The leftconnection member 80L connects the image-pickup/detection unit 10 andthe battery unit 90 on the left side of the user's body (right side inFIG. 1A).

The image-pickup/detection unit 10 is provided with a face directiondetection window 13, a start switch 14, a stop switch 15, an imagepickup lens 16, an LED 17, and microphones 19L and 19R.

The face direction detection window 13 permits transmission of infraredlight projected from infrared LEDs 22 (FIG. 5: infrared irradiationunit) built in the image-pickup/detection unit 10 to detect positions offace parts of the user. The face direction detection window 13 alsopermits transmission of reflected infrared light from the face.

The start switch 14 is used to start an image pickup operation. The stopswitch 15 is used to stop the image pickup operation. The image pickuplens 16 guides light to be picked up to a solid state image sensor 42(FIG. 5) inside the image-pickup/detection unit 10. The LED 17 indicatesa state that the image pickup operation is on-going or a warning.

The microphones 19R and 19L take in peripheral sound. The microphone 19Ltakes in sound of the left side of user's periphery (right side in FIG.1A). The microphone 19R takes in sound of the right side of the user'speriphery (left side in FIG. 1A).

FIG. 1B is a view showing a state where the user wears the camera body1. When the user wears the camera body 1 so that the battery unit 90will come to a user's back side and the image-pickup/detection unit 10will come to the front side of the user's body, theimage-pickup/detection unit 10 is supported while being energized in adirection toward a chest by the left and right connection members 80Land 80R that are respectively connected to the left and right ends ofthe image-pickup/detection unit 10. Thereby, the image-pickup/detectionunit 10 is positioned in front of clavicles of the user. At this time,the face direction detection window 13 is located under a jaw of theuser. An infrared condenser lens 26 shown in FIG. 2E mentioned later isarranged inside the face direction detection window 13. An optical axis(detection optical axis) of the infrared condenser lens 26 is directedto the user's face and is directed to a different direction from anoptical axis (image pickup optical axis) of the image pickup lens 16. Aface direction detection unit 20 (see FIG. 5) including the infraredcondenser lens 26 detects a user's observation direction on the basis ofpositions of face parts. This enables an image pickup unit 40 mentionedlater to pick up an image of an object in the observation direction.

Adjustment of the setting position due to individual difference of abody shape and difference in clothes will be mentioned later. Moreover,since the image-pickup/detection unit 10 is arranged in the front sideof the body and the battery unit 90 is arranged in the back face in thisway, weight of the camera body 1 is distributed, which reduces user'sfatigue and reduces displacement of the camera body 1 due to centrifugalforce caused by movement of the user.

Although the example in which the user wears the camera body 1 so thatthe image-pickup/detection unit 10 will be located in front of theclavicles of the user is described, the embodiment is not limited tothis example. That is, the user may wear the camera body 1 in anyposition of the user's body other than the head as long as the camerabody 1 can detect the user's observation direction and the image pickupunit 40 can pick up an image in an object in the observation direction.

FIG. 1C is a view showing the battery unit 90 viewed from a rear side inFIG. 1A. The battery unit 90 is provided with a charge cable insertingslot 91, adjustment buttons 92L and 92R, and a backbone escape cutout 93in FIG. 1C.

A charge cable (not shown) can be connected to the charge cableinserting slot 91. Batteries 94L and 94R (see FIG. 3A) are chargedthrough the charge cable and electric power is supplied to theimage-pickup/detection unit 10 through the charge cable.

Adjustment buttons 92L and 92R are used to adjust the respective lengthsof the band parts 82L and 82R of the left and right connection members80L and 80R. The adjustment button 92L is used to adjust the left bandpart 82L, and the adjustment button 92R is used to adjust the right bandpart 82R. Although the lengths of the band parts 82L and 82R areindependently adjusted with the adjustment buttons 92L and 92R in theembodiment, the lengths of the band parts 82L and 82R may besimultaneously adjusted with one button.

The backbone escape cutout 93 is formed to escape the backbone of theuser so that the battery unit 90 will not touch the backbone. Since thebackbone escape cutout 93 escapes a convex part of the backbone of thebody, displeasure of wearing is reduced and lateral displacement of thebattery unit 90 is prevented.

FIG. 1D is an external view showing the display apparatus 800 as aportable device according to the first embodiment that is separated fromthe camera body 1. As shown in FIG. 1D, the display apparatus 800 isprovided with an A-button 802, a display unit 803, a B-button 804, anin-camera 805, a face sensor 806, an angular speed sensor 807, and anacceleration sensor 808. Moreover, the display apparatus 800 is providedwith a wireless LAN unit (not shown in FIG. 1D) that enables high-speedconnection with the camera body 1.

The A-button 802 has a function of a power button of the displayapparatus 800. The display apparatus 800 receives an ON/OFF operation bya long press of the A-button 802 and receives a designation of anotherprocess timing by a short press of the A-button 802.

The display unit 803 is used to check an image picked up by the camerabody 1 and can display a menu screen required for setting. In thisembodiment, a transparent touch sensor that is provided on the surfaceof the display unit 803 receives a touch operation to a screen (forexample, a menu screen) that is displaying.

The B-button 804 functions as a calibration button 854 used for acalibration process mentioned later.

The in-camera 805 can pick up an image of a person who is observing thedisplay apparatus 800.

The face sensor 806 detects a face shape and an observation direction ofthe person who is observing the display apparatus 800. A concreteconfiguration of the face sensor 806 is not limited. For example, astructural optical sensor, a ToF (Time of Flight) sensor, and amilliwave radar may be employed.

Since the angular speed sensor 807 is built in the display apparatus800, it is shown by a dotted line as a meaning of a perspective view.Since the display apparatus 800 of this embodiment is also provided witha function of the calibrator mentioned later, a triaxle gyro sensor thatenables detection in X, Y, and Z directions is provided.

The acceleration sensor 808 detects a posture of the display apparatus800. It should be noted that a general smart phone is employed as thedisplay apparatus 800 according to this embodiment. The camera system ofthe embodiment is achieved by adjusting firmware in the smart phone tofirmware of the camera body 1. In the meantime, the camera system of theembodiment can be achieved by adjusting the firmware of the camera body1 to an application and OS of the smart phone as the display apparatus800.

FIG. 2A through FIG. 2F are views describing the image-pickup/detectionunit 10 in detail. In views from FIG. 2A, a component that has the samefunction of a part that has been already described is indicated by thesame reference numeral and its description in this specification isomitted.

FIG. 2A is a front view showing the image-pickup/detection unit 10. Theright connection member 80R has the band part 82R and an angle-holdingmember 81R of hard material that holds an angle with respect to theimage-pickup/detection unit 10. The left connection member 80L has theband part 82L and an angle-holding member 81L similarly.

FIG. 2B is a view showing the shapes of the band parts 82L and 82R ofthe left and right connection members 80L and 80R. In FIG. 2B, the angleholding members 81L and 81R are shown as transparent members in order toshow the shapes of the band parts 82L and 82R. The band part 82L isprovided with a left connecting surface 83L and an electric cable 84that are arranged at the left side of the user's body (right side inFIG. 2B) when the user wears the camera body 1. The band part 82R isprovided with a right connecting surface 83R arranged at the right sideof the user's body (left side in FIG. 2B) when the user wears the camerabody 1.

The left connecting surface 83L is connected with the angle holdingmember 81L, and its sectional shape is an ellipse but is not a perfectcircle. The right connecting surface 83R also has a similar ellipticalshape. The right connecting surface 83R and left connecting surface 83Lare arranged symmetrically in a reverse V-shape. That is, the distancebetween the right connecting surface 83R and the left connecting surface83L becomes shorter toward the upper side from the lower side in FIG.2B. Thereby, since the long axis directions of sections of the left andright connecting surfaces 83L and 83R match the user's body when theuser hangs the camera body 1, the band parts 82L and 82R touch theuser's body comfortably and movement of the image-pickup/detection unit10 in the left-and-right direction and front-and-back direction can beprevented.

The electric cable (a power supply member) 84 is wired inside the bandpart 82L and electrically connects the battery unit 90 and theimage-pickup/detection unit 10. The electric cable 84 connects the powersource of the battery unit 90 to the image-pickup/detection unit 10 ortransfers an electrical signal with an external apparatus.

FIG. 2C is a rear view showing the image-pickup/detection unit 10. FIG.2C shows the side that contacts to the user's body. That is, FIG. 2C isa view viewed from the opposite side of FIG. 2A. Accordingly, thepositional relationship between the right connection member 80R and theleft connection member 80L is contrary to FIG. 2A.

The image-pickup/detection unit 10 is provided with a power switch 11,an image pickup mode switch 12, and chest contact pads 18 a and 18 b atthe back side. The power switch 11 is used to switch ON/OFF of the powerof the camera body 1. Although the power switch 11 of this embodiment isa slide lever type, it is not limited to this. For example, the powerswitch 11 may be a push type switch or may be a switch that isintegrally constituted with a slide cover (not shown) of the imagepickup lens 16.

The image pickup mode switch (a change member) 12 is used to change animage pickup mode, i.e., is used to change a mode in connection with animage pickup operation. In this embodiment, the image pickup mode switch12 is used to select the image pickup mode from among a still imagemode, a video image mode, and a below-mentioned preset mode that is setusing the display apparatus 800. In this embodiment, the image pickupmode switch 12 is a slide lever switch that can select one of “Photo”,“Normal”, and “Pre” shown in FIG. 2C. The image pickup mode shifts tothe still image mode by sliding to “Photo”, shifts to the video imagemode by sliding to “Normal”, and shifts to the preset mode by sliding to“Pre”. It should be noted that the configuration of the image pickupmode switch 12 is not limited to the embodiment as long as a switch canchange the image pickup mode. For example, the image pickup mode switch12 may consist of three buttons of “Photo”, “Normal”, and “Pre”.

The chest contact pads (fixing members) 18 a and 18 b touch the user'sbody when the image-pickup/detection unit 10 is energized to the user'sbody. As shown in FIG. 2A, the image-pickup/detection unit 10 is formedso that a lateral (left-and-right) overall length will become longerthan a vertical (up-and-down) overall length in wearing the camera body1. The chest contact pads 18 a and 18 b are respectively arranged invicinities of right and left ends of the image-pickup/detection unit 10.This arrangement reduces rotational blur in the left-and-right directionduring the image pickup operation of the camera body 1. Moreover, thechest contact pads 18 a and 18 b prevent the power switch 11 and theimage pickup mode switch 12 from touching the user's body. Furthermore,the chest contact pads 18 a and 18 b prevent heat transmission to theuser's body even if the image-pickup/detection unit 10 heats up due to along-time image pickup operation and are used for the adjustment of theangle of the image-pickup/detection unit 10.

FIG. 2D is a top view showing the image-pickup/detection unit 10. Asshown in FIG. 2D, the face direction detection window 13 is provided inthe central part of the top surface of the image-pickup/detection unit10, and the chest contact pads 18 a and 18 b are projected from theimage-pickup/detection unit 10.

FIG. 2E is a view showing a configuration of the face directiondetection unit 20 arranged inside the image-pickup/detection unit 10 andunder the face direction detection window 13. The face directiondetection unit 20 is provided with the infrared LEDs 22 and the infraredcondenser lens 26. The face direction detection unit 20 is also providedwith an infrared LED lighting circuit 21 and an infrared detectiondevice 27 shown in FIG. 5 mentioned later.

The infrared LEDs 22 project infrared light 23 (FIG. 5) toward the user.The infrared condenser lens 26 images reflected light 25 (FIG. 5) fromthe user in projecting the infrared light 23 from the infrared LEDs 22 sonto a sensor (not shown) of the infrared detection device 27.

FIG. 2F is a view showing a state where a user wears the camera body 1viewed from a left side of the user.

An angle adjustment button 85L is provided in the angle holding member81L and is used in adjusting the angle of the image-pickup/detectionunit 10. An angle adjustment button (not shown in FIG. 2F) is providedin the opposite angle holding member 81R in the symmetrical position ofthe angle adjustment button 85L.

Although the angle adjustment buttons are actually visible in FIG. 2A,FIG. 2C, and FIG. 2D, they are omitted to simplify the description.

When moving the angle holding member 81L upward or downward in FIG. 2Fwhile pressing the angle adjustment button 85L, the user can change theangle between the image-pickup/detection unit 10 and the angle holdingmember 81L. The right side is the same as the left side. Moreover,projection angles of the chest contact pads 18 a and 18 b can bechanged. The functions of these two kinds of angle change members (theangle adjustment buttons and chest contact pads) can adjust theimage-pickup/detection unit 10 so as to keep the optical axis of theimage pickup lens 16 horizontally irrespective of individual differenceof a chest position shape.

FIG. 3A, FIG. 3B, and FIG. 3C are views showing details of the batteryunit 90. FIG. 3A is a partially transparent back view showing thebattery unit 90.

As shown in FIG. 3A, the left battery 94L and right battery 94R aresymmetrically mounted inside the battery unit 90 in order to achieveweight balance. In this way, since the left and right batteries 94L and94R are arranged symmetrically with the central part of the battery unit90, the weight balance in the left-and-right direction is achieved andthe position displacement of the camera body 1 is prevented. It shouldbe noted that the battery unit 90 may mount a single battery.

FIG. 3B is a top view showing the battery unit 90. The batteries 94L and94R are shown as the transparent members also in FIG. 3B.

As shown in FIG. 3B, since the batteries 94L and 94R are symmetricallyarranged at both the sides of the backbone escape cutout 93, the usercan wear the battery unit 90 that is relatively heavy without anyburden.

FIG. 3C is a rear view showing the battery unit 90. FIG. 3C is the viewviewed from the side touched to the user's body, i.e., is the viewviewed from the opposite side of FIG. 3A. As shown in FIG. 3C, thebackbone escape cutout 93 is provided in the center along the backboneof the user.

FIG. 4 is a functional block diagram showing the camera body 1.Hereinafter, the process executed by the camera body 1 will be describedroughly using FIG. 4. Details will be mentioned later. As shown in FIG.4, the camera body 1 is provided with the face direction detection unit20, a recording-direction/field-angle determination unit 30, the imagepickup unit 40, an image extraction/development unit 50, a primaryrecording unit 60, a transmission unit 70, and a second controller 111.These functional blocks are achieved by control of an overall controlCPU 101 (FIG. 5) that controls the entire camera body 1.

The face direction detection unit 20 (an observation direction detectionunit) is a functional block executed by the above-mentioned infraredLEDs 22, the infrared detection device 27, etc. The face directiondetection unit 20 estimates an observation direction by detecting theface direction and passes the observation direction to therecording-direction/field-angle determination unit 30.

The recording-direction/field-angle determination unit (a recordingdirection determination unit) 30 determines information about a positionand an area that will be extracted from an image picked up by the imagepickup unit 40 by performing various calculations on the basis of theobservation direction estimated by the face direction detection unit 20.And then, the information is passed to the image extraction/developmentunit 50.

The image pickup unit 40 forms a wide-angle image of the object andpasses the image to the image extraction/development unit 50. The imageextraction/development unit (a development unit) 50 extracts an imagethat the user looks at from the image passed from the image pickup unit40 by using the information passed from therecording-direction/field-angle determination unit 30. Then, the imageextraction/development unit 50 develops the extracted image and passesthe developed image to the primary recording unit 60.

The primary recording unit 60 is a functional block constituted by aprimary memory 103 (FIG. 5) etc., records image information, and passesthe image information to the transmission unit 70 at a required timing.The transmission unit (an image output unit) 70 is wirelessly connectedwith predetermined communication parties, such as the display apparatus(FIG. 1D) 800, a calibrator 850, and a simplified display device 900,and communicates with these.

The display apparatus 800 is connectable to the transmission unit 70through a high-speed wireless LAN (hereinafter referred to as a“high-speed wireless network”). In this embodiment, the high-speedwireless network employs wireless communication corresponding to theIEEE802.11ax (Wi-Fi 6) standard. In the meantime, wireless communicationcorresponding to other standards, such as the Wi-Fi 4 standard and theWi-Fi 5 standard, may be employed. Moreover, the display apparatus 800may be a dedicated apparatus developed for the camera body 1 or may be ageneral smart phone, a tablet terminal, etc.

In addition, the display apparatus 800 may be connected to thetransmission unit 70 through a small-power wireless network, may beconnected through both the high-speed wireless network and small-powerwireless network, or may be connected while switching the networks. Inthis embodiment, large amount data like an image file of a video imagementioned later is transmitted through the high-speed wireless network,and small amount data and data that does not need quick transmission aretransmitted through the small-power wireless network. Although theBluetooth is used for the small-power wireless network in thisembodiment, other short-distance wireless communications, such as theNFC (Near Field Communication), may be employed.

The calibrator 850 performs initial setting and individual setting ofthe camera body 1, and is connectable to the transmission unit 70through the high-speed wireless network in the same manner as thedisplay apparatus 800. Details of the calibrator 850 are mentionedlater. Moreover, the display apparatus 800 may have the function of thecalibrator 850.

The simplified display device 900 is connectable to the transmissionunit 70 only through the small-power wireless network, for example.Although the simplified display device 900 cannot perform communicationof a video image with the transmission units 70 due to time restriction,it can transmit an image pickup start/stop timing and can be used toimage check at a composition check level. Moreover, the simplifieddisplay device 900 may be a dedicated apparatus developed for the camerabody 1 as well as the display apparatus 800 or may be a smart watch.

FIG. 5 is a block diagram showing a hardware configuration of the camerabody 1. Moreover, the configurations and functions described using FIG.1A through FIG. 1C are indicated by the same reference numerals andtheir detailed descriptions will be omitted.

As shown in FIG. 5, the camera body 1 is provided with the overallcontrol CPU 101, power switch 11, image pickup mode switch 12, facedirection detection window 13, start switch 14, stop switch 15, imagepickup lens 16, and LED 17.

The camera body 1 is provided with the infrared LED lighting circuit 21,infrared LEDs 22, infrared condenser lens 26, and infrared detectiondevice 27 that constitute the face direction detection unit 20 (FIG. 4).

Moreover, the camera body 1 is provided with the image pickup unit 40(FIG. 4), which consists of an image pickup driver 41, a solid stateimage sensor 42, and an image signal processing circuit 43, and thetransmission unit 70 (FIG. 4), which consists of a small-power wirelesscommunication unit 71 and high-speed wireless communication unit 72.

Although the camera body 1 has the single image pickup unit 40 in thisembodiment, it may have two or more image pickup units in order to pickup a 3D image, to pick up an image of which a field angle is wider thanan image obtained by a single image pickup unit, or to pick up images indifferent directions.

The camera body 1 is provided with various memories, such as alarge-capacity nonvolatile memory 51, an internal nonvolatile memory102, and the primary memory 103 degree again. Furthermore, the camerabody 1 is provided with an audio processor 104, a speaker 105, avibrator 106, an angular speed sensor 107, an acceleration sensor 108,and various switches 110.

The switches like the power switch 11, which are described using FIG.2C, are connected to the overall control CPU 101. The overall controlCPU 101 controls the entire camera body 1. Therecording-direction/field-angle determination unit 30, imageextraction/development unit 50, and second controller 111 in FIG. 4 areachieved by overall control CPU 101.

The infrared LED lighting circuit 21 controls lighting and extinction ofthe infrared LEDs 22 (FIG. 2E) to control projection of the infraredlight 23 directed to the user from the infrared LEDs 22.

The face direction detection window 13 is constituted by a visible lightcut filter that almost cuts off visible light and sufficiently permitstransmission of the infrared light 23 and its reflected light 25 thatbelong to infrared region. The infrared condenser lens 26 condenses thereflected light 25.

The infrared detection device (an infrared detection unit) 27 has asensor that detects the reflected light 25 condensed by the infraredcondenser lens 26. The sensor converts an image formed by the condensedreflected light 25 into sensor data and passes the sensor data to theoverall control CPU 101.

As shown in FIG. 1B, when the user wears the camera body 1, the facedirection detection window 13 is located under a user's jaw.Accordingly, as shown in FIG. 5, the infrared light 23 projected fromthe infrared LEDs 22 transmits the face direction detection window 13and an infrared irradiation surface 24 near the user's jaw is irradiatedwith the infrared light 23. Moreover, the reflected light 25 reflectedfrom the infrared irradiation surface 24 transmits the face directiondetection window 13 and is condensed by the infrared condenser lens 26onto the sensor in the infrared detection device 27.

The various switches 110 are not shown in FIG. 1A through FIG. 1C. Thevarious switches 110 are used to execute functions that are unrelated tothis embodiment.

The image pickup driver 41 includes a timing generator, generatesvarious timing signals, outputs the timing signals to sections relatedto the image pickup operation, and drives the solid state image sensor42. The solid state image sensor 42 outputs the signal obtained byphotoelectric conversion of the object image formed through the imagepickup lens 16 (FIG. 1A) to the image signal processing circuit 43.

The image signal processing circuit 43 generates the pickup image databy applying a clamp process and an A/D conversion process, etc. to thesignal from the solid state image sensor 42 and outputs the pickup imageto the overall control CPU 101.

The internal nonvolatile memory 102 is constituted by a flash memoryetc. and stores a boot program for the overall control CPU 101 and setvalues of various program modes. In this embodiment, a set value of anobservation visual field (field angle) and a set value of an effectlevel of an image stabilization process are recorded.

The primary memory 103 is constituted by a RAM etc. and temporarilystores processing image data and a calculation result of the overallcontrol CPU 101.

The large-capacity nonvolatile memory 51 stores image data. In thisembodiment, the large-capacity nonvolatile memory 51 is a semiconductormemory that is not detachable. However, the large-capacity nonvolatilememory 51 may be constituted by a detachable storage medium like an SDcard, and may be used together with the internal nonvolatile memory 102.

The small-power wireless communication unit 71 exchanges data with thedisplay apparatus 800, the calibrator 850, and the simplified displaydevice 900 through the small-power wireless network. The high-speedwireless communication unit 72 exchanges data with the display apparatus800 and the calibrator 850 through the high-speed wireless network.

The audio processor 104 processes outside sound (analog signal)collected by the microphones 19L and 19R and generates an audio signal.

In order to notify the user of a state of the camera body 1 and to warnthe user, the LED 17 emits light, the speaker 105 outputs sound, and thevibrator 106 vibrates.

The angular speed sensor 107 uses a gyro etc. and detects movement ofthe camera body 1 as gyro data. The acceleration sensor 108 detects theposture of the image-pickup/detection unit 10.

FIG. 6 is a block diagram showing a hardware configuration of thedisplay apparatus 800. The components that have been described usingFIG. 1D are indicated by the same reference numerals and theirdescriptions will be omitted to simplify the description. As shown inFIG. 6, the display apparatus 800 is provided with a display-apparatuscontroller 801, the A-button 802, the display unit 803, the B-button804, the face sensor 806, the angular speed sensor 807, the accelerationsensor 808, an image signal processing circuit 809, and various switches811.

Moreover, the display apparatus 800 is provided with an internalnonvolatile memory 812, a primary memory 813, a large-capacitynonvolatile memory 814, a speaker 815, a vibrator 816, an LED 817, anaudio processor 820, a small-power wireless communication unit 871, anda high-speed wireless communication unit 872. The above-mentionedcomponents are connected to the display-apparatus controller 801.

The display-apparatus controller 801 is constituted by a CPU andcontrols the display apparatus 800.

The image signal processing circuit 809 bears equivalent functions withthe image pickup driver 41, solid state image sensor 42, and imagesignal processing circuit 43 inside the camera body 1. The image signalprocessing circuit 809 constitutes the in-camera 805 in FIG. 1D togetherwith an in-camera lens 805 a. The display-apparatus controller 801processes the data output from the image signal processing circuit 809.The contents of the process of the data will be mentioned later.

The various switches 811 are used to execute functions that areunrelated to this embodiment.

The angular speed sensor 807 uses a gyro etc. and detects movement ofthe display apparatus 800. The acceleration sensor 808 detects a postureof the display apparatus 800.

The internal nonvolatile memory 812 is constituted by a flash memoryetc. and stores a boot program for the display-apparatus controller 801and set values of various program modes.

The primary memory 813 is constituted by a RAM etc. and temporarilystores processing image data and a calculation result of the imagesignal processing circuit 809. In this embodiment, when a video image isrecording, gyro data detected with the angular speed sensor 107 atpickup time of each frame is stored into the primary memory 813 inassociation with the frame.

The large-capacity nonvolatile memory 51 stores image data of thedisplay apparatus 800. In this embodiment, the large-capacitynonvolatile memory 814 is constituted by a detachable memory like an SDcard. It should be noted that the large-capacity nonvolatile memory 814may be constituted by a fixed memory as with the large-capacitynonvolatile memory 51 in the camera body 1.

In order to notify the user of a state of the display apparatus 800 andto warn the user, the LED 817 emits light, the speaker 815 outputssound, and the vibrator 816 vibrates.

The audio processor 820 processes outside sound (analog signal)collected by the microphones 819L and 819R and generates an audiosignal.

The small-power wireless communication unit 871 exchanges data with thecamera body 1 through the small-power wireless network. The high-speedwireless communication unit 872 exchanges data with the camera body 1through the high-speed wireless network.

The face sensor (a face detection unit) 806 is provided with an infraredLED lighting circuit 821 and infrared LEDs 822, an infrared condenserlens 826, and an infrared detection device 827.

The infrared LED lighting circuit 821 has the function similar to thatof the infrared LED lighting circuit 21 in FIG. 5 and controls lightingand extinction of the infrared LEDs 822 to control projection of theinfrared light 823 directed to the user from the infrared LEDs 822. Theinfrared condenser lens 826 condenses the reflected light 825.

The infrared detection device (an infrared detection unit) 827 has asensor that detects the reflected light 825 condensed by the infraredcondenser lens 826. The sensor converts the condensed reflected light825 into sensor data and passes the sensor data to the display-apparatuscontroller 801.

When the face sensor 806 shown in FIG. 1D is directed to the user, aninfrared irradiation surface 824 that is the entire face of the user isirradiated with the infrared light 823 projected from the infrared LEDs822 as shown in FIG. 6. Moreover, the reflected light 825 reflected fromthe infrared irradiation surface 824 is condensed by the infraredcondenser lens 826 onto the sensor in the infrared detection device 827.

Other functions 830 are functions of a smart phone, such as a telephonefunction, that are not related to the embodiment.

Hereinafter, how to use the camera body 1 and display apparatus 800 willbe described. FIG. 7A is a flowchart schematically showing an imagepickup/recording process according to the first embodiment executed bythe camera body 1 and display apparatus 800.

In order to assist the description, a reference numeral (in FIG. 4 orFIG. 5) of a unit that executes a process in each step is shown on aright side of each step in FIG. 7A. That is, steps S100 through S700 inFIG. 7A are executed by the camera body 1, and steps S800 through S1000in FIG. 7A are executed by the display apparatus 800.

When the power switch 11 is set to ON and power of the camera body 1turns ON, the overall control CPU 101 is activated and reads a bootprogram from the internal nonvolatile memory 102. After that, in a stepS100, the overall control CPU 101 executes a preparation process thatperforms setting of the camera body 1 before an image pickup operation.Details of the preparation process will be mentioned later using FIG.7B.

In a step S200, a face direction detection process that estimates anobservation direction based on a face direction detected by the facedirection detection unit 20 is executed. Details of the face directiondetection process will be mentioned later using FIG. 7C. This process isexecuted at a predetermined frame rate.

In a step S300, the recording-direction/field-angle determination unit30 executes a recording-direction/area determination process. Details ofthe recording-direction/area determination process will be mentionedlater using FIG. 7D. In a step S400, the image pickup unit 40 picks upan image and generates pickup image data.

In the step S500, the image extraction/development unit 50 extracts animage from the pickup image data generated in the step S400 according tothe recording-direction/field-angle information determined in the stepS300 and performs a recording area development process that develops theextracted area. Details of the recording area development process willbe mentioned later using FIG. 7E.

In a step S600, the primary recording unit (an image recording unit) 60executes the primary recording process that stores the image developedin the step S500 into the primary memory 103 as image data. Details ofthe primary recording process will be mentioned later using FIG. 14.

In the step S700, the transmission unit 70 executes a transmissionprocess to the display apparatus 800 that wirelessly transmits the imageprimarily recorded in the step S600 to the display apparatus 800 at adesignated timing. Details of the transfer process to the displayapparatus 800 will be mentioned later using FIG. 16.

The steps from the step S800 is executed by the display apparatus 800.In the step S800, the display-apparatus controller 801 executes anoptical correction process that corrects optical aberrations of theimage transferred from the camera body 1 in the step S700. Details ofthe optical correction process will be mentioned later using FIG. 17.

In a step S900, the display-apparatus controller 801 applies an imagestabilization process to the image of which optical aberrations havebeen corrected in the step S800. Details of the image stabilizationprocess will be mentioned later using FIG. 19.

It should be noted that the order of the step S800 and the step S900 maybe inverted. That is, the image stabilization process may be executed inadvance of the optical correction process.

The display-apparatus controller (a video recording unit) 801 executes asecondary recording process that records the image to which the opticalcorrection process in the step S800 and the image stabilization processin the step S900 have been applied into the large-capacity nonvolatilememory 814 in the step S1000. And then, the display-apparatus controller801 finishes this process.

Next, the processes (subroutines) in the respective steps in FIG. 7Awill be described in detail using FIG. 7B through FIG. 7F and otherdrawings in the order of the processes. FIG. 7B is a flowchart showing asubroutine of the preparation process in the step S100 in FIG. 7A.Hereinafter, this process is described using the components shown inFIG. 2A through FIG. 2F and FIG. 5.

It is determined whether the power switch 11 is ON in a step S101. Theprocess waits when the power is OFF. When the power becomes ON, theprocess proceeds to a step S102. In the step S102, the mode selected bythe image pickup mode switch 12 is determined. As a result of thedetermination, when the mode selected by the image pickup mode switch 12is the video image mode, the process proceeds to a step S103.

In the step S103, various set values of the video image mode are readfrom the internal nonvolatile memory 102 and are stored into the primarymemory 103. Then, the process proceeds to a step S104. The various setvalues of the video image mode include a field-angle set value V_(ang)and an image stabilization level. The field-angle set value V_(ang) ispreset to 90° in this embodiment. The image stabilization level isselected from among “Strong”, “Middle”, and “OFF”.

In the step S104, an operation of the image pickup driver 41 for thevideo image mode is started. And then, the process exits thissubroutine. As a result of the determination in the step S102, when themode selected by the image pickup mode switch 12 is the still imagemode, the process proceeds to a step S106.

In the step S106, various set values of the still image mode are readfrom the internal nonvolatile memory 102 and are stored into the primarymemory 103. Then, the process proceeds to a step S107. The various setvalues of the still image mode include the field-angle set value V_(ang)and then image stabilization level. The field-angle set value V_(ang) ispreset to 45° in this embodiment. The image stabilization level isselected from among “Strong”, “Middle”, and “OFF”.

In the step S107, an operation of the image pickup driver 41 for thestill image mode is started. And then, the process exits thissubroutine.

As the result of the determination in the step S102, when the modeselected by the image pickup mode switch 12 is the preset mode, theprocess proceeds to a step S108. The preset mode is one of the threeimage pickup modes that can be changed by the image pickup mode switch12. In the preset mode, the image pickup mode of the camera body 1 canbe changed by an external device like the display apparatus 800. Thatis, the preset mode is for a custom image pickup operation. Since thecamera body 1 is a compact wearable device, operation switches, asetting screen, etc. for changing advanced set values are not mounted onthe camera body 1. The advanced set values are changed by an externaldevice like the display apparatus 800.

For example, a case where the user would like to change the field anglefrom 90° to 110° while picking up a video image continuously isconsidered. In such a case, the following operations are needed. Sincethe field angle is set to 90° in a regular video image mode, the userperforms the video image pickup operation in the regular video imagemode, once finishes the video image pickup operation, displays thesetting screen on the display apparatus 800, and changes the field angleto 110° on the setting screen. However, the operations to the displayapparatus 800 during a certain event are troublesome.

In the meantime, when the preset mode is preset to a video image pickupoperation at the field angle of 110°, the user can change the fieldangle in the video image pickup operation to 110° immediately by onlysliding the image pickup mode switch 12 to “Pre” after finishing thevideo image pickup operation at the field angle of 90°. That is, theuser is not required to suspend the current operation and to perform theabove-mentioned troublesome operations.

It should be noted that contents of the preset mode may include theimage stabilization level (“Strong”, “Middle”, or “OFF”) and a set valueof voice recognition that is not described in this embodiment inaddition to the field angle.

In the step S108, various set values of the preset mode are read fromthe internal nonvolatile memory 102 and are stored into the primarymemory 103. Then, the process proceeds to a step S109. The various setvalues of the preset mode include the field-angle set value V_(ang) andthe image stabilization level that is selected from among “Strong”,“Middle”, and “OFF”.

In the step S109, an operation of the image pickup driver 41 for thepreset mode is started. And then, the process exits this subroutine.

Hereinafter, the various set values of the video image mode read in thestep S103 will be describe using FIG. 13. FIG. 13 is a view showing amenu screen for setting the various set values of the video image modethat is displayed on the display unit 803 of the display apparatus 800before an image pickup operation of the camera body 1. The componentsthat have been described using FIG. 1D are indicated by the samereference numerals and their descriptions will be omitted. The displayunit 803 has a touch panel function and will be described under thepresumption that it functions by touch operations, such as a swipeoperation.

As shown in FIG. 13, the menu screen includes a preview screen 831, azoom lever 832, a recording start/stop button 833, a switch 834, abattery residue indicator 835, a button 836, a lever 837, and an icondisplay area 838. The user can check the image picked up by the camerabody 1, a zoom amount, and a field angle on the preview screen 831.

The user can change a zoom setting (a field angle) by shifting the zoomlever 832 rightward or leftward. This embodiment describes a case wherethe field-angle set value V_(ang) can be selected from among 45°, 90°,110°, and 130°. In the meantime, the field-angle set value V_(ang) maybe set to a value other than the four values by operating the zoom lever832.

The recording start/stop button 833 is a toggle switch that has both ofthe function of the start switch 14 and the function of the stop switch15. The switch 834 is used to switch “OFF” and “ON” of the imagestabilization process. The battery residue indicator 835 displaysbattery residue of the camera body 1. The button 836 is used to change amode.

The lever 837 is used to set the image stabilization level. Although theimage stabilization level can be set to “Strong” or “Middle” in thisembodiment, another image stabilization level, for example “Weak”, maybe set. Moreover, the image stabilization level may be set steplessly. Aplurality of thumbnail icons for preview are displayed in the icondisplay area 838.

FIG. 7C is a flowchart showing a subroutine of the face directiondetection process in the step S200 in FIG. 7A. Before describing thedetails of this process, a face direction detection method usinginfrared light will be described using FIG. 8A through FIG. 8K.

FIG. 8A is a view showing the visible light image of a user's facelooked at from the position of the face direction detection window 13.The image in FIG. 8A is equivalent to an image picked up by avisible-light image sensor on the assumption that the face directiondetection window 13 permits transmission of visible light and that thevisible-light image sensor is mounted in the infrared detection device27. The image in FIG. 8A includes a neck front part 201 above claviclesof the user, a root 202 of a jaw, a chin 203, and a face 204 including anose.

FIG. 8B is a view showing a case where fluorescent lamps in a roomappear as background in the visible-light image of the user shown inFIG. 8A. The fluorescent lamps 205 around the user appear in thevisible-light image in FIG. 8B. In this way, since various backgroundsappear in a user's image according to a use condition, it becomesdifficult that the face direction detection unit 20 or the overallcontrol CPU 101 cuts out a face image from a visible-light image. In themeantime, although there is a technique that cuts such an image by usingan AI etc., the technique is not suitable for the camera body 1 as aportable device because the overall control CPU 101 is required to havehigh performance.

Accordingly, the camera body 1 of the first embodiment detects a user'sface using an infrared image. Since the face direction detection window13 is constituted by a visible light cut filter, visible light is nottransmitted mostly. Accordingly, an image obtained by the infrareddetection device 27 is different from the images in FIG. 8A and FIG. 8B.

FIG. 8C is a view showing an infrared image obtained by imaging the userand fluorescent lamps as background shown in FIG. 8B onto the sensor ofthe infrared detection device 27 through the face direction detectionwindow 13 in a state where the infrared LEDs 22 are not lightened.

In the infrared image in FIG. 8C, the user's neck and jaw are dark. Inthe meantime, since the fluorescent lamps 205 emit an infrared componentin addition to the visible light, they are slightly bright.

FIG. 8D is a view showing an image obtained by imaging the user andfluorescent lamps as background shown in FIG. 8B onto the sensor of theinfrared detection device 27 through the face direction detection window13 in a state where the infrared LEDs 22 are lightened.

In the image in FIG. 8D, the user's neck and jaw are bright. In themeantime, unlike FIG. 8C, the brightness around the fluorescent lamps205 has not changed. FIG. 8E is a view showing a difference image thatis calculated by subtracting the image in FIG. 8C from the image in FIG.8D. The user's face emerges.

In this way, the overall control CPU (an image obtainment unit) 101obtains the difference image (hereinafter referred to as a face image)by calculating the difference between the image formed on the sensor ofthe infrared detection device 27 in the state where the infrared LEDs 22are lightened and the image formed on the sensor in the state where theinfrared LEDs 22 are not lightened.

The face direction detection unit 20 of this embodiment employs a methodthat obtains a face image by extracting infrared reflection intensity asa two-dimensional image by the infrared detection device 27. The sensorof the infrared detection device 27 employs a configuration similar to ageneral image sensor and obtains a face image frame-by-frame. A verticalsynchronization signal (hereinafter referred to as a V-signal) thatobtains frame synchronization is generated by the infrared detectiondevice 27 and is output to the overall control CPU 101.

FIG. 9 is a timing chart showing timings of lighting and extinction ofthe infrared LEDs 22 and related signals. A V-signal output from theinfrared detection device 27, an H-position of the image signal outputfrom the sensor of the infrared detection device 27, an IR-ON signaloutput to the infrared LED lighting circuit 21 from the overall controlCPU 101, and pickup image data output to the overall control CPU 101from the sensor of the infrared detection device 27 are shown in FIG. 9in the order from the top. The horizontal time axes of these foursignals are identical.

When the V-signal becomes High, timings of the frame synchronization andtimings of lighting and extinction of the infrared LEDs 22 are obtained.FIG. 9 shows a first face image obtainment period t1 and a second faceimage obtainment period t2.

The infrared detection device 27 controls the operation of the sensor sothat the H-position of the image signal will synchronize with theV-signal as shown in FIG. 9. Since the sensor of the infrared detectiondevice 27 employs the configuration similar to a general image sensor asmentioned above and its operation is well-known, a detailed descriptionof the control method is omitted.

The overall control CPU 101 controls switching of the IR-ON signalbetween High and Low in synchronization with the V-signal. Specifically,the overall control CPU 101 outputs the IR-ON signal of Low to theinfrared LED lighting circuit 21 during the period t1 and outputs theIR-ON signal of High to the infrared LED lighting circuit 21 during thesecond period t2.

The infrared LED lighting circuit 21 lightens the infrared LEDs 22 toproject the infrared light 23 to the user during the High period of theIR-ON signal. In the meantime, the infrared LED lighting circuit 21switches the infrared LEDs 22 off during the Low period of the IR-ONsignal.

A vertical axis of the pickup image data indicates a signal intensitythat is a light receiving amount of the reflected light 25. Since theinfrared LEDs 22 are not lightened during the first period t1, noreflected light comes from the user's face and pickup image data asshown in FIG. 8C is obtained. In the meantime, since the infrared LEDs22 are lightened during the second period t2, the reflected light 25comes from the user's face and pickup image data as shown in FIG. 8D isobtained. Accordingly, the signal intensity in the period t2 increasesfrom the signal intensity in the period t1 by the reflected light 25from the user's face.

A face image indicated in the bottom in FIG. 9 is obtained bysubtracting the image pickup data during the first period t1 from theimage pickup data during the second period t2. As a result of thesubtraction, face image data in which only the component of thereflected light 25 from the user's face is extracted is obtained.

FIG. 7C shows the face direction detection process in the step S200 thatincludes the operations described using FIG. 8C through FIG. 8E and FIG.9.

In a step S201, a timing V1 at which the first period t1 starts isobtained when the V-signal output from the infrared detection device 27becomes High. When the timing V1 is obtained, the process proceeds to astep S202. In the step S202, the IR-ON signal is set to Low and isoutput to the infrared LED lighting circuit 21. Thereby, the infraredLEDs 22 are not lightened.

In a step S203, one frame of pickup image data output from the infrareddetection device 27 during the first period t1 is read. The image datais temporarily stored into the primary memory 103 as Frame1.

In a step S204, a timing V2 at which the second period t2 starts isobtained when the V-signal output from the infrared detection device 27becomes High. When the timing V1 is obtained, the process proceeds to astep S205.

In the step S205, the IR-ON signal is set to High and is output to theinfrared LED lighting circuit 21. Thereby, the infrared LEDs 22 arelightened.

In a step S206, one frame of pickup image data output from the infrareddetection device 27 during the second period t2 is read. The image datais temporarily stored into the primary memory 103 as Frame2.

In a step S207, the IR-ON signal is set to Low and is output to theinfrared LED lighting circuit 21. Thereby, the infrared LEDs 22 are notlightened.

In a step S208, Frame1 and Frame2 are read from the primary memory 103,and light intensity Fn of the reflected light 25 from the usercorresponding to the face image shown in FIG. 9 is calculated bysubtracting Frame1 from Frame2. This process is generally called blacksubtraction.

In a step S209, a throat position (a head rotation center) is extractedfrom the light intensity Fn. First, the overall control CPU (a divisionunit) 101 divides the face image into a plurality of distance areas thatwill be described using FIG. 8F on the basis of the light intensity Fn.

FIG. 8F is a view showing a result obtained by adjusting shades of thedifference image shown in FIG. 8E so as to fit with a scale of lightintensity of the reflected light 25 of the infrared light 23 projectedto the face and neck of the user. FIG. 8F shows light intensitydistribution about sections of the face and neck of the user.

The face image on the left side in FIG. 8F shows the light intensitydistribution of the reflected light 25 in the face image shown in FIG.8E by gray steps applied to the respective divided areas. An Xf axis isadded in a direction from the central part of the user's neck toward thechin. In a graph on the right side in FIG. 8F, a horizontal axis showsthe light intensity on the Xf axis of the face image and a vertical axisshows the Xf axis. The light intensity shown by the horizontal axisincreases as going rightward. The face image in FIG. 8F is divided intosix areas (distance areas) 211 through 216 according to the lightintensity.

The area 211 is an area where the light intensity is the strongest andis shown by white among the gray steps. The area 212 is an area wherethe light intensity falls slightly than the area 211 and is shown byquite bright gray among the gray steps. The area 213 is an area wherethe light intensity falls still more than the area 212 and is shown bybright gray among the gray steps.

The area 214 is an area where the light intensity falls still more thanthe area 213 and is shown by middle gray among the gray steps. The area215 is an area where the light intensity falls still more than the area214 and is shown by slightly dark gray among the gray steps.

The area 216 is an area where the light intensity is the weakest and isshown by the darkest gray among the gray steps. The area above the area216 is shown by black showing no light intensity.

The light intensity will be described in detail using FIG. 10A throughFIG. 10D. FIG. 10A through FIG. 10D are views describing movement of theuser's face in the vertical direction and show states observed from theleft side of the user.

FIG. 10A is a view showing a state where the user faces the front. Thereis the image-pickup/detection unit 10 in front of the clavicles of theuser. Moreover, the infrared light 23 of the infrared LEDs 22 irradiatesthe lower part of the user's head from the face direction detectionwindow 13 mounted in the upper portion of the image-pickup/detectionunit 10. A distance Dn from the face direction detection window 13 tothe throat 200 above the clavicles of the user, a distance db from theface direction detection window 13 to the root 202 of the jaw, and adistance Dc from the face direction detection window 13 to the chin 203satisfy a relation of Dn<db<Dc. Since light intensity is in inverseproportion to the square of distance, the light intensity in the imageformed by the reflected light 25 on the sensor of the infraredirradiation surface 24 becomes gradually weak in the order of the throat200, the root 202 of the jaw, and the chin 203. Moreover, since thedistance from the face direction detection window 13 to the face 204including the nose is still longer than the distance Dc, the lightintensity in the image corresponding to the face 204 becomes stillweaker. That is, in the case as shown in FIG. 10A, the image having thelight intensity distribution shown in FIG. 8F is obtained.

It should be noted that the configuration of the face directiondetection unit 20 is not limited to the configuration shown in thisembodiment as long as the face direction of the user can be detected.For example, an infrared pattern may be projected from the infrared LEDs(an infrared pattern irradiation unit) 22, and the sensor (an infraredpattern detection unit) of the infrared detection device 27 may detectthe infrared pattern reflected from an irradiation target. In this case,it is preferable that the sensor of the infrared detection device 27 isconstituted by a structural optical sensor. Moreover, the sensor of theinfrared detection device 27 may be a sensor (an infrared phasecomparison unit) that compares the phase of the infrared light 23 andthe phase of the reflected light 25. For example, a ToF sensor may beemployed.

Next, the extraction of the throat position in the step S209 in FIG. 7Cwill be described using FIG. 8G. A left image in FIG. 8G is obtained bysuperimposing the reference numerals denoting the parts of the user'sbody shown in FIG. 10A, a double circle showing the throat position, anda black circle showing the chin position on FIG. 8F.

The white area 211 corresponds to the throat 200 (FIG. 10A), the quitebright gray area 212 corresponds to the neck front part 201 (FIG. 10A),and the bright gray area 213 corresponds to the root 202 of the jaw(FIG. 10A). Moreover, the middle gray area 214 corresponds to the chin203 (FIG. 10A), and the slightly dark gray area 215 corresponds to a liplocated in the lower part of the face 204 (FIG. 10A) and a face lowerpart around the lip. Furthermore, the darkest gray area 216 correspondsto the nose located in the center of the face 204 (FIG. 10A) and a faceupper part around the nose.

Since the difference between the distances db and Dc is relatively smallas compared with the differences between the other distances from theface direction detection window 13 to other parts of the user as shownin FIG. 10A, the difference between the reflected light intensities inthe bright gray area 213 and the middle gray area 214 is also small.

In the meantime, since the distance Dn is the shortest distance amongthe distances from the face direction detection window 13 to the partsof the user as shown in FIG. 10A, the reflection light intensity in thewhite area 211 corresponding to the throat 200 becomes the strongest.

Accordingly, the overall control CPU (a setting unit) 101 determinesthat the area 211 corresponds to the throat 200 and its periphery, andthen, sets the position 206 (indicated by the double circle in FIG. 8G),which is located at the center in the lateral direction and is thenearest to the image-pickup/detection unit 10, as the position of thehead rotation center (hereinafter referred to as a throat position 206).The processes up to the moment are the contents performed in the stepS209 in FIG. 7C.

Next, the extraction of the chin position in the step S210 in FIG. 7Cwill be described using FIG. 8G. In the image in FIG. 8G, the middlegray area 214 that is brighter than the area 215 corresponding to theface lower part including the lip of the face 204 includes the chin. Agraph on the right side in FIG. 8G shows that the light intensity fallssharply in the area 215 adjacent to the area 214 because the change rateof the distance from the face direction detection window 13 becomeslarge. The overall control CPU 101 determines that the brighter area 214adjacent to the area 215 in which the light intensity falls sharply is achin area. Furthermore, the overall control CPU 101 calculates(extracts) the position (indicated by the black circle shown in FIG.8G), which is located at the center in the lateral direction in the area214 and is the farthest from the throat position 206, as the chinposition 207.

For example, FIG. 8H and FIG. 8I show changes in directing the face tothe right. FIG. 8H is a view showing a difference image calculated bythe similar method as FIG. 8E in directing the user's face to the right.FIG. 8I is a view showing a result obtained by adjusting shades of thedifference image in FIG. 8H so as to fit with a scale of lightintensities of reflected components of the infrared light projected tothe face and neck of the user and by superimposing the double circleshowing the throat position 206 as the position of the head rotationcenter and the black circle showing the chin position 207 r.

Since the user's face is directed to the right, the area 214 moves to anarea 214 r shown in FIG. 8I that is located in the left side when it islooked up from the image-pickup/detection unit 10. The area 215corresponding to the face lower part including the lip in the face 204also moves to an area 215 r that is located in the left side when it islooked up from the image-pickup/detection unit 10.

Accordingly, the overall control CPU 101 determines that the brighterarea 214 r adjacent to the area 215 r in which the light intensity fallssharply is the chin area. Furthermore, the overall control CPU 101calculates (extracts) the position (indicated by the black circle shownin FIG. 8I), which is located at the center in the lateral direction inthe area 214 r and is the farthest from the throat position 206, as thechin position 207 r.

After that, the overall control CPU 101 finds a moving angle θr thatshows the rotational movement to the right from the chin position 207 inthe image in FIG. 8G to the chin position 207 r in FIG. 8I around thethroat position 206. As shown in FIG. 8I, the moving angle θr is anangle of movement of the user's face in a lateral direction.

The angle of face (hereinafter, referred to as a face angle) of the userin the lateral direction is calculated in the step S210 from the chinposition detected by the infrared detection device 27 of the facedirection detection unit (a three-dimensional detection sensor) 20.

Next, detection of the face directed upward will be described. FIG. 10Bis a view showing a state that the user directs the face horizontally.FIG. 10C is a view showing a state that the user directs the face upwardby 33° from the horizontal direction.

The distance from the face direction detection window 13 to the chin 203is Ffh in FIG. 10B, and the distance from the face direction detectionwindow 13 to the chin 203 u is Ffu in FIG. 10C. Since the chin 203 umoves upwards together with the face, the distance Ffu becomes longerthan the distance Ffh as shown in FIG. 10C.

FIG. 8J is a view showing an image of the user who directs the faceupward by 33° from the horizontal direction viewed from the facedirection detection window 13. Since the user directs the face upward asshown in FIG. 10C, the face 204 including the lip and nose cannot beseen from the face direction detection window 13 located under theuser's jaw. The chin 203 and its neck side are seen. FIG. 8K showsdistribution of the light intensity of the reflected light 25 inirradiating the user in the state shown in FIG. 10C with the infraredlight 23. An image on the left side in FIG. 8K is a view showing aresult obtained by adjusting shades of the difference image calculatedby the same method as FIG. 8E so as to fit with a scale of lightintensities of reflected components of the infrared light projected tothe face and neck of the user and by superimposing the double circleshowing the throat position 206 and the black circle showing the chinposition 207 u. Two graphs in FIG. 8K show density changes of the leftimage. The left graph is equivalent to the graph in FIG. 8F and theright graph is equivalent to the graph in FIG. 8G.

Six areas 211 u, 212 u, 213 u, 214 u, 215 u, and 216 u divided accordingto the light intensity in FIG. 8K are indicated by adding “u” to thereference numerals of the same light intensity areas shown in FIG. 8F.Although the light intensity of the user's chin 203 is included in themiddle gray area 214 in FIG. 8F, it shifts to the black side and isincluded in the slightly dark gray area 215 u in FIG. 8K. In this way,since the distance Ffu is longer than the distance Ffh as shown in FIG.10C, the infrared detection device 27 can detect that the lightintensity of the reflected light 25 from the chin 203 is weakened ininverse proportion to the square of distance.

Next, detection of the face directed downward will be described. FIG.10D is a view showing a state that the user directs the face downward by22° from the horizontal direction. In FIG. 10D, a distance from the facedirection detection window 13 to the chin 203 d is Ffd.

Since the chin 203 u moves downward together with the face, the distanceFfd becomes shorter than the distance Ffh as shown in FIG. 10D and thelight intensity of the reflected light 25 at the chin 203 becomesstronger.

Returning back to FIG. 7C, in a step S211, the overall control CPU (adistance calculation unit) 101 calculates the distance from the chinposition to the face direction detection window 13 on the basis of thelight intensity of the chin position detected by the infrared detectiondevice 27 of the face direction detection unit (three-dimensionaldetection sensor) 20. A face angle in the vertical direction is alsocalculated on the basis of this.

In a step S212, the overall control CPU 101 stores the face angle θh inthe lateral direction (a first detecting direction) obtained in the stepS210 and the face angle θv in the vertical direction (a second detectingdirection) obtained in the step S211 into the primary memory 103 as athree-dimensional observation direction vi (“i” is arbitrary referencenumeral) of the user. For example, when the user is observing the frontcenter, the face angle θh in the lateral direction is 0° and the faceangle θv in the vertical direction is 0°. Accordingly, the observationdirection vo in this case is represented by vector information (0°, 0°).Moreover, when the user is observing in a 45-degree-right direction, theobservation direction vr is represented by vector information (45°, 0°).

Although the face angle in the vertical direction is calculated bydetecting the distance from the face direction detection window 13 inthe step S211, the face angle may be calculated by another method. Forexample, change of the face angle may be calculated by comparing changelevels of the light intensity of the chin 203. That is, the change ofthe face angle may be calculated by comparing a gradient CDh of thereflected light intensity from the root 202 of the jaw to the chin 203in the graph in FIG. 8G with a gradient CDu of the reflected lightintensity from the root 202 of the jaw to the chin 203 in the graph inFIG. 8K.

FIG. 7D is a flowchart showing a subroutine of therecording-direction/area determination process in the step S300 in FIG.7A. Before describing details of this process, a superwide-angle imagethat is subjected to determine a recording direction and a recordingarea in this embodiment will be described first using FIG. 11A.

In the camera body 1 of this embodiment, the image pickup unit 40 picksup a superwide-angle image of the periphery of theimage-pickup/detection unit 10 using the superwide-angle image pickuplens 16. An image of an observation direction can be obtained byextracting a part of the superwide-angle image.

FIG. 11A is a view showing a target visual field 125 set in asuperwide-angle image picked up by the image pickup unit 40 in a casewhere the user faces the front. As shown in FIG. 11A, a pixel area 121that can be picked up by the solid state image sensor 42 is arectangular area. Moreover, an effective projection area (apredetermined area) 122 is an area of a circular half-celestial sphereimage that is a fish-eye image projected on the solid state image sensor42 by the image pickup lens 16. The image pickup lens 16 is adjusted sothat the center of the pixel area 121 will match the center of theeffective projection area 122.

The outermost periphery of the circular effective projection area 122shows a position where a visual field angle is 180°. When the user islooking at the center in both the vertical and horizontal directions, anangular range of the target visual field 125 that is picked up andrecorded becomes 90° (a half of the visual field angle) centered on thecenter of the effective projection area 122. It should be noted that theimage pickup lens 16 of this embodiment can also introduce light outsidethe effective projection area 122 and can project light within themaximum visual field angle (about 192°) to the solid state image sensor42 to form a fish-eye image. However, the optical performance fallsgreatly in the area outside the effective projection area 122. Forexample, resolution falls extremely, light amount falls, and distortionincreases. Accordingly, in this embodiment, an image of an observationdirection is extracted as a recording area only from the inside of theimage (hereinafter referred to as a superwide-angle image, simply)projected on the pixel area 121 within a half-celestial sphere imagedisplayed on the effective projection area 122.

Since the size of the effective projection area 122 in the verticaldirection is larger than the size of the short side of the pixel area121, the upper and lower ends of the image in the effective projectionarea 122 are out of the pixel area 121 in this embodiment. However, therelationship between the areas is not limited to this. For example, theoptical system may be designed so that the entire effective projectionarea 122 will be included in the pixel area 121 by changing theconfiguration of the image pickup lens 16.

Invalid pixel areas 123 are parts of the pixel area 121 that are notincluded in the effective projection area 122. The target visual field125 shows an area of an image of a user's observation direction thatwill be extracted from the superwide-angle image. In the example shownin FIG. 11A, the target visual field 125 is prescribed by left, right,upper, and lower field angles each of which is 45° (the visual fieldangle is 90°) centered on the observation direction. In the example ofFIG. 11A, since the user faces the front, the center of the targetvisual field 125 (the observation direction vo) matches the center ofthe effective projection area 122.

The superwide-angle image shown in FIG. 11 includes an A-object 131 thatis a child, a B-object 132 that shows steps that the child (A-object) istrying to climb, and a C-object 133 that is locomotive-type playgroundequipment.

Next, details of the recording-direction/area determination process inthe step S300 in FIG. 7A will be described. FIG. 7D shows therecording-direction/area determination process executed in order toextract an image of an observation direction from the superwide-angleimage described using FIG. 11A. Hereinafter, this process is describedusing FIG. 12A through FIG. 12G that show concrete examples of thetarget visual field 125.

In a step S301, a field-angle set value V_(ang) that is set in advanceis obtained by reading from the primary memory 103. In this embodiment,the internal nonvolatile memory 102 stores all the available fieldangles (45°, 90°, 110°, and 130°) as field-angle set values V_(ang). Theimage extraction/development unit 50 extracts an image of an observationdirection in an area defined by the field-angle set value V_(ang) fromthe superwide-angle image. Moreover, the field-angle set value V_(ang)included in the various set values read from the internal nonvolatilememory 102 in one of the steps S103, S106, and S108 in FIG. 7B isestablished and is being stored in the primary memory 103.

Moreover, in the step S301, the observation direction vi determined inthe step S212 is determined as the recording direction, an image in thetarget visual field 125 of which the center is designated by theobservation direction vi and of which an area is defined by the obtainedfield-angle set value V_(ang) is extracted from the superwide-angleimage, and the extracted image is stored into the primary memory 103.

For example, when the field-angle set value V_(ang) is 90° and theobservation direction vo (vector information (0°, 0°)) is detectedthrough the face direction detection process (FIG. 7C), the targetvisual field 125 of which the center matches the center O of theeffective projection area 122 and of which the angular widths in thehorizontal and vertical directions are 90° (FIG. 11A) is established.FIG. 11B is a view showing the image in the target visual field 125extracted from the superwide-angle image in FIG. 11A. That is, theoverall control CPU (a relative position setting unit) 101 sets theangle of the face direction detected by the face direction detectionunit 20 to the observation direction vi that is the vector informationshowing the relative position of the target visual field 125 withrespect to the superwide-angle image.

In the case of the observation direction vo, since the influence of theoptical distortion caused by the image pickup lens 16 can be disregardedmostly, the shape of the established target visual field 125 is almostidentical to the shape of a target visual field 125 o (FIG. 12A) afterconverting the distortion in a step S303 mentions later. Hereinafter, atarget visual field after applying the distortion conversion in the caseof the observation direction vi is called a target visual field 125 i.

In a step S302, an image stabilization level that is set in advance isobtained by reading from the primary memory 103. In this embodiment, theimage stabilization level included in the various set values read fromthe internal nonvolatile memory 102 in one of the steps S103, S106, andS108 in FIG. 7B is established and is being stored in the primary memory103.

Moreover, in the step S302, an image-stabilization-margin pixel numberP_(is) is set on the basis of the obtained image stabilization level. Inthe image stabilization process, an image following in a directionopposite to a blur direction is obtained according to a blur amount ofthe image-pickup/detection unit 10. Accordingly, in this embodiment, animage stabilization margin required for the image stabilization isestablished around the target visual field 125 i.

Moreover, in this embodiment, a table that keeps values of theimage-stabilization-margin pixel number P_(is) in association withrespective image stabilization levels is stored in the internalnonvolatile memory 102. For example, when the image stabilization levelis “middle”, “100 pixels” that is the image-stabilization-margin pixelnumber P_(is) corresponding to the level “middle” is read from theabove-mentioned table. And then, an image stabilization margin of whichwidth is 100 pixels is established around the target visual field.

FIG. 12E is a view showing an example that gives an image stabilizationmargin corresponding to a predetermined image stabilization level aroundthe target visual field 125 o shown in FIG. 12A. The followingdescription assumes that the image stabilization level is “middle”(i.e., the image-stabilization-margin pixel number P_(is) is “100pixels”).

As shown by a dotted line in FIG. 12E, an image stabilization margin 126o of which the width is “100 pixels” that is theimage-stabilization-margin pixel number P_(is) is established around thetarget visual field 125 o.

FIG. 12A and FIG. 12E show the case where the observation direction vimatches the center O (the optical axis center of the image pickup lens16) of the effective projection area 122 for simplification of thedescription. In the meantime, when the observation direction vi isdirected to a periphery of the effective projection area 122, conversionis required to reduce influence of optical distortion.

In the step S303, the shape of the target visual field 125 establishedin the step S301 is corrected in consideration of the observationdirection vi and the optical property of the image pickup lens 16 toconvert the distortion and generate the target visual field 125 i.Similarly, the image-stabilization-margin pixel number P_(is) set in thestep S302 is also corrected in consideration of the observationdirection vi and the optical property of the image pickup lens 16.

For example, the user's observation direction shall be a right directionby 45° from the center o and the field-angle set value V_(ang) shall be90°. In this case, the observation direction vr (vector information(45°, 0°)) is determined in the step S212 and the target visual field125 is established as a range of 45° in each of left, right, upper, andlower directions centering on the observation direction vr. Furthermore,the target visual field 125 is corrected to the target visual field 125r shown in FIG. 12B in consideration of the optical property of theimage pickup lens 16.

As shown in FIG. 12B, the target visual field 125 r becomes wider towardthe periphery of the effective projection area 122. And the position ofthe observation direction vr approaches inside a little from the centerof the target visual field 125 r. This is because the optical design ofthe image pickup lens 16 in this embodiment is close to that of astereographic projection fish-eye lens. It should be noted that contentsof the correction depend on the optical design of the image pickup lens16. If the image pickup lens 16 is designed as an equidistant projectionfish-eye lens, an equal-solid-angle projection fish-eye lens, or anorthogonal projection fish-eye lens, the target visual field 125 iscorrected according to its optical property.

FIG. 12F is a view showing an example that gives an image stabilizationmargin 126 r corresponding to the same image stabilization level“middle” of the image stabilization margin in FIG. 12E around the targetvisual field 125 r shown in FIG. 12B. The image stabilization margin 126o (FIG. 12E) is established at the left, right, upper, and lower sidesof the target visual field 125 o with the fixed width of 100 pixels thatis the image-stabilization-margin pixel number P_(is) corresponding tothe level “middle”. As compared with this, theimage-stabilization-margin pixel number P_(is) of the imagestabilization margin 126 r (FIG. 12F) is corrected to increase towardthe periphery of the effective projection area 122.

In this way, the shape of the image stabilization margin required forthe image stabilization around the target visual field 125 r is alsocorrected as with the target visual field 125 r so that the correctionamount will increase toward the periphery of the effective projectionarea 122 as shown by the image stabilization margin 126 r in FIG. 12F.This is also because the optical design of the image pickup lens 16 inthis embodiment is close to that of a stereographic projection fish-eyelens. It should be noted that contents of the correction depend on theoptical design of the image pickup lens 16. If the image pickup lens 16is designed as an equidistant projection fish-eye lens, anequal-solid-angle projection fish-eye lens, or an orthogonal projectionfish-eye lens, the image stabilization margin 126 r is correctedaccording to its optical property.

The process executed in the step S303 that switches successively theshapes of the target visual field 125 and its image stabilization marginin consideration of the optical property of the image pickup lens 16 isa complicated process. Accordingly, in this embodiment, the process inthe step S303 is executed using a table that keeps shapes of the targetvisual field 125 i and its image stabilization margin for everyobservation direction vi stored in the internal nonvolatile memory 102.It should be noted that the overall control CPU 101 may have a computingequation depending on the optical design of the image pickup lens 16. Insuch a case, the overall control CPU 101 can calculate an opticaldistortion value using the computing equation.

In a step S304, a position and size of an image recording frame arecalculated. As mentioned above, the image stabilization margin 126 irequired for the image stabilization is established around the targetvisual field 125 i. However, when the position of the observationdirection vi is close to the periphery of the effective projection area122, the shape of the image stabilization margin becomes considerablyspecial as shown by the image stabilization margin 126 r, for example.

The overall control CPU 101 can extract an image in such aspecial-shaped area and apply the development process to the extractedimage. However, it is not general to use an image that is notrectangular in recording as image data in the step S600 or intransmitting image data to the display apparatus 800 in the step S700.Accordingly, in the step S304, the position and size of the imagerecording frame 127 i of a rectangular shape that includes the entireimage stabilization margin 126 i are calculated.

FIG. 12F shows the image recording frame 127 r that is calculated in thestep S304 to the image stabilization margin 126 r by an alternate longand short dash line. In a step S305, the position and size of the imagerecording frame 127 i that are calculated in the step S304 are recordedinto the primary memory 103.

In this embodiment, an upper-left coordinate (Xi, Yi) of the imagerecording frame 127 i in the superwide-angle image is recorded as theposition of the image recording frame 127 i, and a lateral width WXi anda vertical width WYi that start from the coordinate (Xi, Yi) arerecorded as the size of the image recording frame 127 ii. For example, acoordinate (Xr, Yr), a lateral width WXr, and a vertical width WYr ofthe image recording frame 127 r shown in FIG. 12F are recorded in thestep S305. It should be noted that the coordinate (Xi, Yi) is a XYcoordinate of which the origin is a predetermined reference point,specifically the optical center of the image pickup lens 16.

When the image stabilization margin 126 i and the image recording frame127 i have been determined in this way, the process exits thissubroutine shown in FIG. 7D.

In the description so far, the observation directions of which thehorizontal angle is 0°, such as the observation direction v0 (the vectorinformation (0°, 0°)) and the observation direction vr (the vectorinformation (45°, 0°)), have been described for simplifying thedescription of the complicated optical distortion conversion. In themeantime, an actual observation direction vi of the user is arbitrary.Accordingly, the recording area development process executed in a casewhere the horizontal angle is not 0° will be described hereinafter.

For example, when the field-angle set value V_(ang) is 90° and theobservation direction vm is (−42°, −40°), a target visual field 125 mappears as shown in FIG. 12C. Moreover, when the field-angle set valueV_(ang) is 45° and the observation direction vm is (−42°, −40°), atarget visual field 128 m, which is smaller than the target visual field125 m, appears as shown in FIG. 12D. Furthermore, an image stabilizationmargin 129 m and an image recording frame 130 m are established aroundthe target visual field 128 m as shown in FIG. 12G.

Since the process in the step S400 is a fundamental image pickupoperation and employs a general sequence of the image pickup unit 40,its detailed description is omitted. It should be noted that the imagesignal processing circuit 43 in the image pickup unit 40 in thisembodiment also performs a process that converts signals of an inherentoutput form (standard examples: MIPI, SLVS) output from the solid stateimage sensor 42 into pickup image data of a general sensor readingsystem.

When the video image mode is selected by the image pickup mode switch12, the image pickup unit 40 starts recording in response to a press ofthe start switch 14. After that, the recording is finished when the stopswitch 15 is pressed. In the meantime, when the still image mode isselected by the image pickup mode switch 12, the image pickup unit 40picks up a static image every time when the start switch 14 is pressed.

FIG. 7E is a flowchart showing a subroutine of the recording-areadevelopment process in the step S500 in FIG. 7A. In a step S501, Rawdata of the entire area of the pickup image data (superwide-angle image)generated by the image pickup unit 40 in the step S400 is obtained andis input into an image capturing unit called a head unit (not shown) ofthe overall control CPU 101.

In the next step S502, the part within the image recording frame 127 iis extracted from the superwide-angle image obtained in the step S501 onthe basis of the coordinate (Xi, Yi), lateral width WXi, and verticalwidth WYi that are recorded into the primary memory 103 in the stepS305. After the extraction, a crop development process (FIG. 7F)consisting of steps S503 through S508 is executed only to the pixelswithin the image stabilization margin 126 i. This can reduce acalculation amount significantly as compared with a case where thedevelopment process is executed to the entire area of thesuperwide-angle image read in the step S501. Accordingly, calculationtime and electric power consumption can be reduced.

As shown in FIG. 7F, when the video image mode is selected by the imagepickup mode switch 12, the processes of the steps S200 and S300 and theprocess of step S400 are executed in parallel by the same frame rate ordifferent frame rates. Whenever the Raw data of the entire area of oneframe generated by the image pickup unit 40 is obtained, the cropdevelopment process is executed on the basis of the coordinate (Xi, Yi),lateral width WXi, and vertical width WYi that are recorded in theprimary memory 103 at that point.

When the crop development process is started to the pixels within theimage stabilization margin 126 i, color interpolation that interpolatesdata of color pixels arranged in the Bayer arrangement is executed inthe step S503.

After that, a white balance is adjusted in a step S504 and a colorconversion is executed in a step S505. In a step S506, gamma correctionthat corrects gradation according to a gamma correction value set upbeforehand is performed. In a step S507, edge enhancement is performedcorresponding to an image size.

In the step S508, the image data is converted into a format that can bestored primarily by applying processes like compression. The convertedimage data is stored into the primary memory 103. After that, theprocess exits the subroutine. Details of the data format that can bestored primarily will be mentioned later.

The order of the processes of the steps S503 through S508 executedduring the crop development process may be changed according to theproperty of the camera system. A part of the processes may be omitted.The order and presences of the processes of the steps S503 through S508do not restrict the present disclosure.

Moreover, when the video image mode is selected, the processes of thesteps S200 through S500 are repeatedly executed until the recording isfinished.

According to this process, the calculation amount is significantlyreduced as compared with a case where the development process isexecuted to the entire area read in the step S501. Accordingly, aninexpensive and low-power consumption microcomputer can be employed asthe overall control CPU 101. Moreover, heat generation in the overallcontrol CPU 101 is reduced and the life of the battery 94 becomeslonger.

Moreover, in order to reduce a control load on the overall control CPU101, the optical correction process (the step S800 in FIG. 7A) and theimage stabilization process (the step S900 in FIG. 7A) to the image arenot executed by the camera body 1 in this embodiment. These processesare executed by the display-apparatus controller 801 after transferringthe image to the display apparatus 800. Accordingly, if only data of apartial image extracted from a projected superwide-angle image istransferred to the display apparatus 800, neither the optical correctionprocess nor the image stabilization process can be executed. That is,since the data of the extracted image does not include positioninformation that will be substituted to a formula of the opticalcorrection process and will be used to refer the correction table of theimage stabilization process, the display apparatus 800 cannot executethese processes correctly. Accordingly, in this embodiment, the camerabody 1 transmits correction data including information about anextraction position of an image from a superwide-angle image togetherwith data of the extracted image to the display apparatus 800.

When the extracted image is a still image, since the still image datacorresponds to the correction data one-to-one, the display apparatus 800can execute the optical correction process and image stabilizationprocess correctly, even if these data are separately transmitted to thedisplay apparatus 800. In the meantime, when the extracted image is avideo image, if the video image data and the correction data areseparately transmitted to the display apparatus 800, it becomesdifficult to determine correspondence between each frame of the videoimage data and the correction data. Particularly, when a clock rate ofthe overall control CPU 101 in the camera body 1 slightly differs from aclock rate of the display-apparatus controller 801 in the displayapparatus 800, the synchronization between the overall control CPU 101and the display-apparatus controller 801 will be lost during the videoimage pickup operation for several minutes. This may cause a defect thatthe display-apparatus controller 801 corrects a frame with correctiondata different from the corresponding correction data.

Accordingly, in this embodiment, when transmitting data of an extractedvideo image to the display apparatus 800, the camera body 1 gives itscorrection data appropriately to the data of the video image.Hereinafter, the method is described.

FIG. 14 is a flowchart showing the subroutine of the primary recordingprocess in the step S600 in FIG. 7A. Hereinafter, this process will bedescribed by also referring to FIG. 15. FIG. 14 shows the process of acase where the video image mode is selected by the image pickup modeswitch 12. When the still image mode is selected, this process startsfrom a step S601 and is finished after a process of a step S606.

In a step S601 a, the overall control CPU 101 reads an image of oneframe to which the processes in steps S601 through S606 have not beenapplied from among the video image developed in the recording areadevelopment process (FIG. 7E). Moreover, the overall control CPU (ametadata generation unit) 101 generates correction data that is metadataof the read frame.

In the step S601, the overall control CPU 101 attaches the informationabout the extraction position of the image of the frame read in the stepS601 a to the correction data. The information attached in this step isthe coordinate (Xi, Yi) of the image recording frame 127 i obtained inthe step S305. It should be noted that the information attached in thisstep may be the vector information that shows the observation directionvi.

In a step S602, the overall control CPU (an optical-correction-valueobtainment unit) 101 obtains an optical correction value. The opticalcorrection value is the optical distortion value set up in the stepS303. Alternatively, the optical correction value may be amarginal-light-amount correction value or a diffraction correction valuecorresponding to the lens optical property.

In a step S603, the overall control CPU 101 attaches the opticalcorrection value used for the distortion conversion in the step S602 tothe correction data.

In a step S604, the overall control CPU 101 Determines whether the imagestabilization mode is effective. Specifically, when the imagestabilization mode is “Middle” or “Strong”, it is determined that theimage stabilization mode is effective and the process proceeds to a stepS605. In the meantime, when the image stabilization mode set up inadvance is “OFF”, it is determined that the image stabilization mode isnot effective and the process proceeds to the step S606. The reason whythe step S605 is skipped when the image stabilization mode is “OFF” isbecause the calculation data amount of the overall control CPU 101 andthe data amount of the wireless communication are reduced and the powerconsumption and heat generation of the camera body 1 can be reduced byskipping the step S605. Although the reduction of the data used for theimage stabilization process is described, the data about themarginal-light-amount correction value or the data about the diffractioncorrection value obtained as the optical correction value in the stepS602 may be reduced.

Although the image stabilization mode is set up by the user's operationto the display apparatus 800 in advance in this embodiment, the mode isset up as a default setting of the camera body 1. Moreover, when thecamera system is configured to switch the effectiveness of imagestabilization process after transferring image data to the displayapparatus 800, the process may directly proceed to the step S605 fromthe step S603 by omitting the step S604.

In the step S605, the overall control CPU (a moving amount detectionunit) 101 attaches the image stabilization mode, which is obtained inthe step S302, and the gyro data, which is obtained during the pickupoperation of the video image in association with the frame that is readfrom the primary memory 813 in the step S601 a, to the correction data.

In the step S606, the overall control CPU 101 updates a video file 1000(FIG. 15) by data obtained by encoding the image data and correctiondata. The image data corresponds to the frame read in the step S606. Thecorrection data includes the various data attached in the steps S601through S605. It should be noted that when a first frame of the videoimage is read in the step S601 a, the video file 1000 is generated inthe step S606.

In a step S607, the overall control CPU 101 determines whether all theframes of the video image developed by the recording area developmentprocess (FIG. 7E) have been read. When not all the frames have beenread, the process returns to the step S601 a. In the meantime, when allthe frames have been read, the process exits this subroutine. Thegenerated video file 1000 is stored into the internal nonvolatile memory102. The video image may be stored into the large-capacity nonvolatilememory 51 too in addition to the primary memory 813 and the internalnonvolatile memory 102. Moreover, the transmission process (the stepS700 in FIG. 7A) that transfers the generated image file 1000 to thedisplay apparatus 800 immediately is executed. The image file 1000 maybe stored into the primary memory 813 after transferring it to thedisplay apparatus 800.

In this embodiment, the encoding means to combine the image data and thecorrection data into one file. At that time, the image data may becompressed or the data file that is combined by the image data andcorrection data may be compressed.

FIG. 15 is a view showing a data structure of the video file 1000. Thevideo file 1000 consists of a header part 1001 and a frame part 1002.The frame part 1002 consists of frame data sets each of which consistsof an image of each frame and corresponding frame metadata. That is, theframe part 1002 includes frame data sets of the number of the totalframes of the video image.

In this embodiment, frame metadata is information obtained by encodingcorrection data to which an extraction position (in-image positioninformation), an optical correction value, and gyro data are attached ifneeded. However, the frame metadata is not limited to this. Aninformation amount of the frame metadata may be changed. For example,other information may be added to the frame metadata according to theimage pickup mode selected by the image pickup mode switch 12.Alternatively, a part of the information in the frame metadata may bedeleted.

A head address and offset values to the respective frame data sets ofthe frame are recorded in the header part 1001. Alternatively, metadatalike the time and size corresponding to the video file 1000 may bestored in the header part 1001.

In the primary recording process (FIG. 14), the video file 1000 istransferred to the display apparatus 800. The video file 100 includesdata sets each of which consists of a frame of the video image developedby recording area development process (FIG. 7E) and its metadata.Accordingly, even when the clock rate of the overall control CPU 101 inthe camera body 1 slightly differs from the clock rate of thedisplay-apparatus controller 801 in the display apparatus 800, thedisplay-apparatus controller 801 certainly applies the correctionprocess to the video image developed in the camera body 1.

Although the optical correction value is included in the frame metadatain this embodiment, the optical correction value may be given to theentire image.

FIG. 16 is a flowchart of the subroutine of the transmission process tothe display apparatus 800 in the step S700 in FIG. 7A. FIG. 16 shows theprocess of a case where the video image mode is selected by the imagepickup mode switch 12. It should be noted that when the still image modeis selected, this process starts from a process in a step S702.

In a step S701, it is determined whether the image pickup process (thestep S400) of the video image by the image pickup unit 40 is finished orit is under recording. When the video image is recording (during thevideo image pickup operation), the recording area development process(the step S500) for each frame and the update of the image file 1000(the step S606) in the primary recording process (step S600) areexecuted sequentially. Since a power load of wireless transmission islarge, if the wireless transmission is performed during the video imagepickup operation in parallel, the battery 94 is needed to have largebattery capacity or a new measure against heat generation is needed.Moreover, from a viewpoint of arithmetic capacity, if the wirelesstransmission is performed during the video image pickup operation inparallel, an arithmetic load will become large, which needs to prepare ahigh-specification CPU as the overall control CPU 101, increasing thecost. In view of these points, in this embodiment, the overall controlCPU 101 proceeds with the process to a step S702 after the video imagepickup operation is finished (YES in the step S701), and establishes thewireless connection with the display apparatus 800. In the meantime, ifthe camera system of the embodiment has a margin in the electric powersupplied from the battery 94 and a new measure against heat generationis unnecessary, the overall control CPU 101 may beforehand establish thewireless connection with the display apparatus 800 when the camera body1 is started or when the video image pickup operation is not yetstarted.

In the step S702, the overall control CPU 101 establishes the connectionwith the display apparatus 800 through the high-speed wirelesscommunication unit 72 in order to transfer the video file 1000 havingmuch data volume to the display apparatus 800. It should be noted thatthe small-power wireless communication unit 71 is used for transmissionof a low-resolution image for checking a field angle to the displayapparatus 800 and is used for exchange of various set values with thedisplay apparatus 800. In the meantime, the small-power wirelesscommunication unit 71 is not used for transfer of the video file 1000because a transmission period becomes long.

In a step S703, the overall control CPU 101 transfers the video file1000 to the display apparatus 800 through the high-speed wirelesscommunication unit 72. When the transmission is finished, the overallcontrol CPU 101 proceeds with the process to a step S704. In the stepS704, the overall control CPU 101 closes the connection with the displayapparatus 800 and exits this subroutine.

The case where one image file including the image of all the frames ofone video image has been described so far. In the meantime, if therecording period of the video image is longer than several minutes, thevideo image may be divided by a unit time into a plurality of imagefiles. When the video file has the data structure shown in FIG. 15, evenif one video image is transferred to the display apparatus 800 as aplurality of image files, the display apparatus 800 becomes available tocorrect the video image without the timing gap with the correction data.

FIG. 17 is a flowchart showing a subroutine of the optical correctionprocess in the step S800 in FIG. 7A. Hereinafter, this process will bedescribed by also referring to FIG. 18A through FIG. 18E. As mentionedabove, this process is executed by the display-apparatus controller 801of the display apparatus 800.

In the step S801, the display-apparatus controller (a video filereception unit) 801 first receives the video file 1000 from the camerabody 1 transferred in the transmission process (the step S700) to thedisplay apparatus 800. After that, the display-apparatus controller (afirst extraction unit) 801 obtains the optical correction valuesextracted from the received video file 1000.

In the next step S802, the display-apparatus controller (a secondextraction unit) 801 obtains an image (an image of one frame obtained bythe video image pickup operation) from the video file 1000.

In a step S803, the display-apparatus controller (a frame imagecorrection unit) 801 performs the optical correction process to correctoptical aberrations of the image obtained in the step S802 with theoptical correction value obtained in the step S801, and stores thecorrected image into the primary memory 813. An image area(extraction-development area) that is narrower than the development area(target visual field 125 i) determined in the step S303 is extractedfrom the image obtained in the step S802 and the optical correctionprocess is applied to the extracted image area.

FIG. 18A through FIG. 18F are views for describing a process of applyingdistortion correction in the step S803 in FIG. 17. FIG. 18A is a viewshowing a position of an object 1401 at which the user looks with anaked eye in picking up an image. FIG. 18B is a view showing an image ofthe object 1401 formed on the solid state image sensor 42.

FIG. 18C is a view showing a development area 1402 in the image in FIG.18B. The development area 1402 is the extraction-development areamentioned above.

FIG. 18D is a view showing an extraction-development image obtained byextracting the image of the development area 1402. FIG. 18E is a viewshowing an image obtained by correcting distortion in theextraction-development image in FIG. 18D. Since an extraction process isperformed in correcting distortion of the extraction-development image,a field angle of the image shown in FIG. 18E becomes still smaller thanthat of the extraction-development image shown in FIG. 18D.

FIG. 19 is a flowchart showing a subroutine of the image stabilizationprocess in the step S900 in FIG. 7A. Hereinafter, this process will bedescribed by also referring to FIG. 18F. As mentioned above, thisprocess is executed by the display-apparatus controller 801 of thedisplay apparatus 800.

In a step S901, the display-apparatus controller 801 obtains gyro dataof a current frame, gyro data of a previous frame, and a blur amountV_(n−1) ^(Det), which is calculated in a below-mentioned step S902 forthe previous frame, from the frame metadata of the video file 1000.After that, a rough blur amount V_(n) ^(Pre) is calculated from thesepieces of information. It should be noted that a current frame in thisembodiment is a frame under processing and that a previous frame is animmediately preceding frame.

In the step S902, the display-apparatus controller 801 calculates a fineblur amount V_(n) ^(Det) from the video file. A blur amount is detectedby calculating a moving amount of a feature point from a previous frameto a current frame.

A feature point can be extracted by a known method. For example, amethod using a luminance information image that is generated byextracting only luminance information of an image of a frame may beemployed. This method subtracts an image that shifts the originalluminance information image by one or several pixels from the originalluminance information image. A pixel of which an absolute value ofdifference exceeds a threshold is extracted as a feature point.Moreover, an edge extracted by subtracting an image generated byapplying a high-pass filter to the above-mentioned luminance informationimage from the original luminance information image may be extracted asa feature point.

Differences are calculated multiple times while shifting the luminanceinformation images of the current frame and previous frame by one orseveral pixels. The moving amount is obtained by calculating a positionat which the difference at the pixel of the feature point diminishes.

Since a plurality of feature points are needed as mentioned later, it ispreferable to divide each of the images of the present frame andprevious frame into a plurality of blocks and to extract a feature pointfor each block. A block division depends on the number of pixels andaspect ratio of the image. In general, 12 blocks of 4*3 or 54 blocks of9*6 is preferable. When the number of blocks is too small, trapezoidaldistortion due to a tilt of the image pickup unit 40 of the camera body1 and rotational blur around the optical axis, etc. cannot be correctedcorrectly. In the meantime, when the number of blocks is too large, asize of one block becomes small, which shortens a distance betweenadjacent feature points, causing error. In this way, the optimal numberof blocks is selected depending on the pixel number, ease of detectionof feature points, a field angle of an object, etc.

Since the obtainment of the moving amount needs a plurality ofdifference calculations while shifting the luminance information imagesof the current frame and previous frame by one or several pixels, thecalculation amount increases. Since the moving amount is actuallycalculated on the basis of the rough blur amount V_(n) ^(Pre) anddeviation (the number of pixels) therefrom, the difference calculationsare performed only near the rough blur amount, which can significantlyreduce the calculation amount.

Next, in a step S903, the display-apparatus controller 801 performs theimage stabilization process using the fine blur amount V_(n) ^(Det)obtained in the step S902. And then, the process exits this subroutine.It should be noted that Euclidean transformation and affinetransformation that enable rotation and parallel translation, andprojective transformation that enables keystone correction are known asthe method of the image stabilization process.

Although the Euclidean transformation can correct movements in theX-axis direction and Y-axis direction and rotation, it cannot correctblur caused by camera shake of the image pickup unit 40 of the camerabody 1 in a front-back direction or directions of pan and tilt.Accordingly, in this embodiment, the image stabilization process isexecuted using the affine transformation that enables correction ofskew. The affine transformation from a coordinate (x, y) of the featurepoint used as criteria to a coordinate (x′, y′) is expressed by thefollowing formula 100.

$\begin{matrix}{\begin{pmatrix}x^{\prime} \\y^{\prime} \\1\end{pmatrix} = {\begin{pmatrix}a & b & c \\d & e & f \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}x \\y \\1\end{pmatrix}}} & {{Formula}100}\end{matrix}$

Affine coefficients of a 3×3 matrix of the formula 100 are computable ifdeviations of at least three feature points are detected. However, whenthe detected feature points are mutually near or are aligned on astraight line, the image stabilization process becomes inaccurate inareas distant from the feature points or distant from the straight line.Accordingly, as for the feature point detected, it is preferable toselect what is in a far distance mutually and does not get on a straightline. Accordingly, when a plurality of feature points are detected,mutually near feature points are excluded and remaining feature pointsare normalized by a least square method.

FIG. 18F is a view showing an image obtained by applying the imagestabilization process in the step S903 to the distortion-corrected imageshown in FIG. 18E. Since the extraction process is performed inexecuting the image stabilization process, a field angle of the imageshown in FIG. 18F becomes smaller than that of the image shown in FIG.18E.

It is available to obtain a high quality image of which blur iscorrected by performing such an image stabilization process. In theabove, the series of operations executed by the camera body 1 anddisplay apparatus 800 that are included in the camera system of thisembodiment have been described.

When the user selects the video image mode by the image pickup modeswitch 12 after turning the power switch 11 ON and observes the frontwithout turning the face in the vertical and horizontal directions, theface direction defection unit 20 detects the observation direction vo(vector information (0°, 0°)) as shown in FIG. 12A. After that, therecording-direction/field-angle determination unit 30 extract the image(FIG. 11B) in the target visual field 125 o shown in FIG. 12A from thesuperwide-angle image projected onto the solid state image sensor 42.

After that, when the user starts observing the child (A-object 131) inFIG. 11A, for example, without operating the camera body 1, the facedirection detection unit 20 detects the observation direction vm (vectorinformation (−42°, −40°)) as shown in FIG. 11C. After that, therecording-direction/field-angle determination unit 30 extracts the image(FIG. 11C) in the target visual field 125 m from the superwide-angleimage picked up by the image pickup unit 40.

In this way, the display apparatus 800 applies the optical correctionprocess and image stabilization process to the extracted image of theshape depending on the observation direction in the steps S800 and S900.Thereby, even if the specification of the overall control CPU 101 of thecamera body 1 is low, the significantly distorted image in the targetvisual field 125 m (FIG. 11C) is converted into the image around thechild (A-object 131) of which the blur and distortion are corrected asshown in FIG. 11D. That is, after the user turns the power switch 11 ONand selects the mode with the image pickup mode switch 12, the user isable to obtain an image of the own observation direction, even if theuser does not touch the camera body 1.

Hereinafter, the preset mode will be described. Since the camera body 1is a compact wearable device, operation switches, a setting screen, etc.for changing advanced set values are not mounted on the camera body 1.Accordingly, in this embodiment, the advanced set values of the camerabody 1 is changed using the setting screen (FIG. 13) of the displayapparatus 800 as an external device.

For example, a case where the user would like to change the field anglefrom 90° to 45° while picking up a video image continuously isconsidered. In such a case, the following operations are needed. Sincethe field angle is set to 90° in a regular video image mode, the userperforms the video image pickup operation in the regular video imagemode, once finishes the video image pickup operation, displays thesetting screen on the display apparatus 800, and changes the field angleto 45° on the setting screen. However, this operation to the displayapparatus 800 during the continuous image pick-up operation istroublesome and an image that the user wants to pick up may be missed.

In the meantime, when the preset mode is preset to a video image pickupoperation at the field angle of 45°, the user can change to a zoom-upvideo image pickup operation at the field angle of 45° immediately byonly sliding the image pickup mode switch 12 to “Pre” after finishingthe video image pickup operation at the field angle of 90°. That is, theuser is not required to suspend the current image pickup operation andto perform the above-mentioned troublesome operations.

The contents of the preset mode may include the image stabilizationlevel (“Strong”, “Middle”, or “OFF”) and a set value of voicerecognition that is not described in this embodiment in addition to thefield angle.

For example, when the user switches the image pickup mode switch 12 fromthe video image mode to the preset mode while continuously observing thechild (A-object 131) in the previous situation, the field-angle setvalue V_(ang) is changed from 90° to 45°. In this case, therecording-direction/field-angle determination unit 30 extracts the imagein the target visual field 128 m shown by a dotted frame in FIG. 11Efrom the superwide-angle image picked up by the image pickup unit 40.

Also in the preset mode, the optical correction process and imagestabilization process are performed in the display apparatus 800 in thesteps S800 and S900. Thereby, even if the specification of the overallcontrol CPU 101 of the camera body 1 is low, the zoom-up image aroundthe child (A-object 131) of which the blur and distortion are correctedas shown in FIG. 11F is obtained. Although the case where thefield-angle set value V_(ang) is changed from 90° to 45° in the videoimage mode has been described, the process in the still image mode issimilar. Moreover, a case where the field-angle set value V_(ang) of avideo image is 90° and the field-angle set value V_(ang) of a staticimage is 45° is also similar.

In this way, the user is able to obtain the zoom-up image that picks upthe own observation direction by just switching the mode with the imagepickup mode switch 12 of the camera body 1.

Although the case where the face direction detection unit 20 and theimage pickup unit 40 are integrally constituted in the camera body 1 isdescribed in this embodiment, the configuration is not limited to thisas long as the face direction detection unit 20 is worn on the user'sbody other than the head and the image pickup unit 40 is worn on theuser's body. For example, the image-pickup/detection unit 10 of thisembodiment can be worn on a shoulder or an abdomen. However, when theimage pickup unit 40 is worn on a right shoulder, an object of the leftside is obstructed by the head. In such a case, it is preferable that aplurality of image pickup units are worn on places including a rightshoulder. Moreover, when the image pickup unit 40 is worn on an abdomen,spatial parallax occurs between the image pickup unit 40 and the head.In such a case, it is preferable to perform a correction calculation ofthe observation direction that compensate such parallax as described ina third embodiment.

Hereinafter, a second embodiment will be described. In the secondembodiment, a method to calibrate individual difference and adjustmentdifference of a user who wears the camera body 1 is described in detailusing FIG. 20A through FIG. 23E.

This embodiment is described as a derivation from the first embodimentbasically. Accordingly, configurations of the camera system in thesecond embodiment that are identical to the configurations of the camerasystem in the first embodiment are indicated by the same referencenumerals and duplicated descriptions are omitted. A differentconfiguration will be described by adding details.

A user who wears the camera body 1 has individual difference andadjustment difference, such as a physique, a tilt angle of periphery ofa neck to which the camera body 1 is worn, a state of a cloth like acollar in wearing, and adjustment states of the band parts 82L and 82R.Accordingly, the optical axis center of the image pickup lens 16 of thecamera body 1 and the visual field center in a state (henceforth anatural state) where the user faces the front do not coincide usually.It is preferable for a user to match a center of an extraction recordingarea (target visual field 125) to a visual field center of the user in acurrent posture or operation rather than to match the center of therecording area to the optical axis center of the image pickup lens 16 ofthe camera body 1.

Moreover, there is individual difference not only in a visual fieldcenter of a user in the natural state but also in a visual field centerdepending on a head direction (up, down, right, left, or slants) and ina motion space of a head. Accordingly, individual difference alsogenerates in the relationship between the face direction (observationdirection) detected by the face direction detection unit 20 and thecenter position (hereinafter referred to as a visual field centerposition) of the target visual field 125 established according to theobservation direction. Accordingly, a calibration operation thatassociates a face direction to a visual field center position is needed.

Usually, the calibration operation is preferably performed as a part ofthe preparation process (the step S100) in FIG. 7A. Although it isestimated of performing the calibration operation at the first start-upof the camera body 1 usually, the calibration operation may be performedwhen a predetermined time elapses after the previous calibration or whenthe position of the camera body 1 to the user is changed from theposition at the previous calibration. The calibration operation may beperformed when the face direction detection unit 20 becomes impossibleto detect a user's face. Moreover, when it is detected that the userdetaches the camera body 1, the calibration operation may be performedat the time when the user again wears the camera body 1. In this way, itis preferable to perform the calibration operation suitably at a timingwhen it is determined that the calibration is needed to use the camerabody 1 appropriately.

FIG. 20A and FIG. 20B are the views showing details of the calibrator850 used for the calibration process according to the second embodiment.In this embodiment, the calibrator 850 shall combine the function of thedisplay apparatus 800.

The calibrator 850 includes a positioning index 851 and calibrationbutton 854 in addition to the A-button 802, display unit 803, in-camera805, face sensor 806, and angular speed sensor 807 that are thecomponents of the display apparatus 800 shown in FIG. 1D. The B-button804 provided in the first embodiment is not illustrated in FIG. 20Abecause it is not used in this embodiment and is replaceable with thecalibration button 854 as mentioned later.

FIG. 20A shows a case where the positioning index 851 is a specificpattern displayed on the display unit 803. FIG. 20B shows a case wherethe external appearance of the calibrator 850 is used as the positioningindex. In the case of FIG. 20B, a positioning index center 852 mentionedlater is calculated from the information about the contour of thecalibrator 850.

It should be noted that the positioning index is not limited to theexamples of FIG. 20A and FIG. 20B. For example, the positioning indexmay be separated from the calibrator 850. The positioning index may beanything as long as its size is easily measured and its shape issuitable to be looked at by the user. For example, the positioning indexmay be a lens cap of the image pickup lens 16 or the and a charge unitfor the camera body 1. Anyway, since a fundamental way of thinking inthe calibration operation is common, the calibrator 850 shown in FIG.20A is exemplified and is mainly described hereinafter.

It should be noted that the calibrator 850 in this embodiment shallcombine the function of the display apparatus 800. Moreover, thecalibrator 850 may be a dedicated device, a general smart phone, or atablet terminal, for example.

The positioning index 851 is an index displayed on the display unit 803of the calibrator 850. A lateral width L851 a and vertical width L851 bof the positioning index 185 and the positioning index center 852 can becalculated. Since the user directs the face toward the vicinity of thecentral part of the positioning index 851 in the calibration processmentioned later, the positioning index 851 is preferably shaped so as tobe caught at the visual field center. In FIG. 20A, the positioning index851 is shown by a circle in which a cross and a small black circle atthe center of the cross are arranged. However, the shape of thepositioning index 851 is not limited to this shape. Otherwise, thepositioning index may be a rectangle, a triangle, a star-shaped figure,or an illustration of a character.

The positioning index 851 is picked up by the image pickup unit 40 ofthe camera body 1. The display-apparatus controller (a positioncalculation unit and a distance calculation unit) 801 calculates adistance between the image-pickup/detection unit 10 and the calibrator850 and a positional coordinate of the positioning index 851 appeared inan image area on the basis of the pickup image. The calibrator 850equipped with the function of the display apparatus 800 performs thesecalculations in this embodiment. If the calibrator 850 does not combinethe function of the display apparatus 800, these calculations areperformed by the overall control CPU 101 of the camera body 1.

The angular speed sensor 807 can measure movement of the calibrator 850.On the basis of the measurement value of the angular speed sensor 807,the display-apparatus controller 801 calculates later-mentioned movementinformation that shows the position and posture of the calibrator 850.

The calibration button 854 is pressed when the user directs the facetoward the vicinity of the central part of the positioning index 851.Although the calibration button 854 is a touch button displayed on thetouch-sensitive display unit 803 in FIG. 20A, the A-button 802 or theB-button 804 may function as the calibration button.

Next, the calibration process executed in extracting an image from asuperwide-angle image picked up by the image pickup unit 40 according toa user's face direction and in applying the image process to theextracted image will be described in detail using a flowchart in FIG.21.

FIG. 21 is the flowchart showing the calibration process according tothe second embodiment executed by the camera body (a first calibrationunit) 1 and calibrator 805.

In order to assist the description, a step in which the camera body 1 orthe calibrator 850 receives a user's instruction is included in a frameof which an operation subject is the user. Moreover, in FIG. 21, a stepexecuted by the display-apparatus controller 801 of the calibrator 850in response to the user's instruction is included in a frame of which anoperation subject is the calibrator 850. Similarly, in FIG. 21, a stepexecuted by the overall control CPU 101 of the camera body 1 in responseto the user's instruction is included in a frame of which an operationsubject is the camera body 1.

Specifically, the operation subject of steps S3104 and S3108 in FIG. 21is the camera body 1. And the operation subject of steps S3101, S3105,and S3106 is the user. Moreover, the calibrator 850 is the operationsubject of steps S3102, S3103, S3106 a, S3107, S3107 b, and S3110.

In this process, when the power of the calibrator 850 is not ON, theuser turns the power of the calibrator 850 ON by operating the A-button802 in a step S3101. Similarly, when the power of the camera body 1 isnot ON, the user turns ON the power of the camera body 1 by switchingthe power switch 11 to ON. After that, the user establishes a connectionbetween the calibrator 850 and the camera body 1. When this connectionis established, the display-apparatus controller 801 and the overallcontrol CPU 101 enter a calibration mode, respectively.

Moreover, in the step S3101, the user wears the camera body 1, andadjusts the lengths of the band parts 82L and 82R and the angle of thecamera body 1 so that the camera body 1 will be arranged in a suitableposition and the image-pickup/detection unit 10 can pick up an image.

In a step S3102, the display-apparatus controller (a first display unit)801 displays the positioning index 851 on the display unit 803.

In the next step S3103, the display-apparatus controller 801 notifiesthe user of a designation position at which the user should hold thecalibrator 850 by an instruction display 855 (FIG. 22A). In thisembodiment, five positions including front, upper right, lower right,upper left, and lower left are designated as the designation positionsin order. The designation positions may be set to other positions aslong as the calibration is available.

In a step S3104, the overall control CPU 101 activates the image pickupunit 40 so as to enable an image pickup operation and activates the facedirection detection unit 20 so as to enable detection of a user's facedirection. In a step S3105, the user holds the calibrator 850 at thedesignation position notified in the step S3103.

In the next step S3106, the user directs the face in the direction ofthe positioning index 851 to match a user's visual field center with thepositioning index 851 and presses the calibration button 854 whilemaintaining the position of the calibrator 850 at the designationposition.

In a step S3106 a, the display-apparatus controller (a second displayunit) 801 determines whether the user looks at the positioning indexcenter 852 of the positioning index 851, i.e., determines whether theuser's visual field center matches the positioning index center 852.When it is determined that the user looks at the positioning indexcenter 852 (YES in the 53106 a), the display-apparatus controller 801notifies the user of start of the calibration for the designationposition by the instruction display 855 in a step S3107 and redisplaysthe calibration button 854. When the determination result in the stepS3106 a is NO, the user repeats the process from the step S3105.

When the user presses the calibration button 854 in the step S3107 a,the display-apparatus controller 801 transmits a calibration instructionto the camera body 1 in a step S3107 b.

In a step S3108, the overall control CPU (an obtainment/detection unit)101 obtains a superwide-angle image including the positioning index 851picked up by the image pickup unit 40 and detects a face direction bythe face direction detection unit 20 in response to the calibrationinstruction from the calibrator 850. After that, the overall control CPU(a generation unit) 101 calculates positional coordinate informationabout the positioning index center 852 in the obtained superwide-angleimage and generates the information showing the relationship between thecalculated positional coordinate information and the detected facedirection.

Hereinafter, the details of the process in the steps S3103 through S3108will be described using FIG. 22A through FIG. 22F. FIG. 22A through FIG.22F are views for describing the calibration operation for the frontdirection of the user. The calibration operation is performed so thatthe center position of the target visual field 125 in the image pickedup by the image pickup unit 40 of the camera body 1 will match thevisual field center position of the user in the natural state.

FIG. 22A is a view showing a screen displayed on the display unit 803 ofthe calibrator 850 in the step S3103 in FIG. 21 during the calibrationoperation for the front direction of the user.

As shown in FIG. 22A, the positioning index 851 and the instructiondisplay 855 that indicates a position at which the user should locatethe positioning index 851 are displayed on the display unit 803 of thecalibrator 850.

The instruction display 855 is a character string that instructs theuser to locate the positioning index 851 at the visual field center ofthe user in directing the face to the front. It should be noted that theinstruction displayed as the instruction display 855 is not restrictedto the character string. For example, the instruction may be displayedby another method using an illustration, a picture, a moving image, orthe like. Moreover, the instruction display 855 like what is called ageneral tutorial may be displayed first and the positioning index 851may be displayed after that.

FIG. 22B is a view showing a state where the user holds the calibrator850 in the front according to the instruction displayed as theinstruction display 855 in FIG. 22A.

In a step S3105, the user holds the calibrator 850 in the frontaccording to the instructions displayed as the instruction display 855in FIG. 22A. Then, in a step S3106, the user holds the calibrator 850 sothat the positioning index 851 will match the visual field center of theuser in directing the face to the front, and the user presses thecalibration button 854 (FIG. 22A). In response to the press of thecalibration button 854, the determination in the step S3106 a isperformed. The concrete procedure of this determination method will bementioned later. When the determination result in the step S3106 a isYES, the display-apparatus controller 801 changes the instructiondisplay 855 shown in FIG. 22A to a notification of “Calibration forFront Direction is Started” and displays the calibration button 854.

Then, the user presses the calibration button 854 after confirming thechange of the instruction display 855 shown in FIG. 22A to thenotification of “Calibration for Front Direction is Started” (a stepS3107 a). In response to the press of the calibration button 854, acalibration instruction is transmitted to the camera body 1 in a stepS3107 b. And the image pickup unit 40 obtains a pickup image in a stepS3108.

FIG. 22C is a schematic view showing the entire superwide-angle imagethat is caught by the image pickup lens 16 in the state of FIG. 22B.FIG. 22D is a schematic view showing an image obtained by correctingaberrations of the superwide-angle image shown in FIG. 22C.

Moreover, in response to the press of the calibration button 854 by theuser in the state of FIG. 22B, the face direction detection unit 20obtains a face direction in the step S3108.

FIG. 22E is a schematic view showing a face direction image that isrecorded by the face direction detection unit 20 in the step S3108 inFIG. 21 during the calibration operation for the front direction of theuser.

As described in the first embodiment using FIG. 8G through FIG. 8K, theface direction detection unit 20 calculates the angles in the lateraland vertical directions of the face using the distances and angles ofthe chin positions 207, 207 r, and 207 u with respect to the throatposition 206. However, since the distances and angles of the chinpositions 207, 207 r, and 207 u with respect to the throat position 206also have the individual difference and adjustment difference due to theuser's physique etc. mentioned above as with the image center, they arenot fixed. Accordingly, in this embodiment, the relationship between thechin position and the throat position 206 at the time of pressing thecalibration button 854 is defined as a value of a case where the userputs the visual field center in the front. This enables correctcalculation of the user's face direction irrespective of the individualdifference and adjustment difference.

Returning back to FIG. 21, in a step S3109, the overall control CPU 101determines whether the calibration for the front direction is prepared.That is, it is determined whether the information required to calculatethe chin position 207, throat position 206, and positioning index center852 has been obtained.

At this time, when the obtainment of the required information is notcompleted, it is determined that the calibration is not prepared (NO inthe step S3109), and the operations from the step S3102 are repeated soas to obtain deficient information among the required information. Whenthe obtainment of the required information is not completed, not all theoperations from the step S3102 are necessary. Only the operations toobtain the deficient information may be performed again.

The determination in the step S3106 a is performed using the face sensor806 or in-camera 805 mounted in the calibrator 850. Hereinafter, theconcrete procedure of this determination method will be described usinga case where the calibration operation for the front direction isperformed using the in-camera 805. Although a case using the face sensor806 is different from the case using the camera in the dimension ofinformation (two-dimensional information or three-dimensionalinformation), a fundamental way of thinking is common. Accordingly,detailed description of the case using the face sensor 806 is omitted.When the face sensor 806 is used in the determination in the step S3106a, the face direction detection unit 20 of the camera body 1 does notperform the face detection that irradiates the user with the infraredlight 23 during a period when the user is irradiated with the infraredlight 823 from the face sensor 806. This aims to prevent interference ofthe infrared lights 23 and 823.

First, when the user presses the calibration button 854 in FIG. 22A inthe step S3106, the display-apparatus controller 801 obtains anin-camera image 858 (FIG. 22F) with the user in it by picking up animage with the in-camera (a face detection unit) 805. Furthermore, thedisplay-apparatus controller 801 detects the position information aboutthe neck front part 201, chin 203, face 204 including a nose, andimage-pickup/detection unit 10 (the image pickup unit 40) from theobtained in-camera image 858.

The display-apparatus controller (a determination unit) 101 determineswhether the user is looking at the positioning index center 852 of thepositioning index 851 at the visual field center in the step S3106 ausing the position information detected from the in-camera image 858.

As a result of the determination, when it is determined that the user islooking in a different direction, the display-apparatus controller 801displays a message indicating that the correct information cannot beobtained as the instruction display 855. This can instruct the user toperform the calibration operation again.

The display-apparatus controller 801 can determine that the correctcalibration operation cannot be performed using the in-camera image 858when the image-pickup/detection unit 10 tilts beyond a certain angle orwhen the face direction detection window 13 is blocked or is dirty. Insuch a case, the display-apparatus controller 801 may display themessage indicating that the correct information cannot be obtained asthe instruction display 855.

Furthermore, it is also able to obtain information required for parallaxcorrection mentioned later in a fifth embodiment using the in-cameraimage 858 obtained in the step S3106 a and the superwide-angle imageobtained in the step S3108.

Specifically, the information about the size (the lateral width L851 aand vertical width L851 b) of the positioning index 851 is transmittedto the camera body 1 beforehand from the calibrator 850 before thepositioning index 851 is picked up by the image pickup unit 40 in thestep S3108. Thereby, the overall control CPU 101 can calculate thedistance between the image-pickup/detection unit 10 and the positioningindex 851 by using the information about the size of the positioningindex 851 and the image of the positioning index 851 appeared in thesuperwide-angle image obtained in the step S3108. Since the positioningindex 851 is included in the calibrator 850 that is the same housing asthe in-camera 805 and the calibrator 850 is directly faced to the userin FIG. 22B, the distance between the in-camera 805 and theimage-pickup/detection unit 10 is equal to the distance between thepositioning index 851 and the image-pickup/detection unit 10.

Similarly, information about the size of the image-pickup/detection unit10 is transmitted to the calibrator 850 beforehand from the camera body1 before the in-camera image shown in FIG. 22F is picked up by thein-camera 805 in the step S3106 a. Thereby, the display-apparatuscontroller (a vertical distance calculation unit) 801 can estimate avertical distance 5070 between the optical axis center the image pickuplens 16 and a view position of the user by using the information aboutthe size of the image-pickup/detection unit 10 and the image of theimage-pickup/detection unit 10 appeared in the in-camera image 858 shownin FIG. 22F. In addition, the display-apparatus controller 801 canestimate a distance 2071 between the image pickup lens 16 and the chin203 of the user. The distance 2071 may be a distance between the facedirection detection window 13 and the chin 203.

In order that the face direction detection unit 20 calculates the throatposition 206 and chin position of the user, it is necessary to separatethe user's face from the face direction detection window 13 at adistance more than a certain distance according to design of the facedirection detection unit 20. Accordingly, this estimated result can beemployed as one of determination conditions in determining whether theface direction detection unit 20 is able to detect the face directioncorrectly.

Returning back to FIG. 21, the overall control CPU 101 proceeds with theprocess to the step S3110 when determining that the required informationis obtained and that the preparation of the calibration for the frontdirection is completed.

In the step S3110, the display-apparatus controller (the firstcalibration unit) 801 calculates information required to offset theextraction center position so as to absorb the individual difference andadjustment difference and offsets the extraction center position on thebasis of the information.

Details of the calculation in the step S3110 will be described asfollows. If the user is in an ideal state according to design values andthe camera body 1 is worn ideally, a center 856 of the superwide-angleimage obtained in the step S3108 shown in FIG. 22C should be almostcoincident with the positioning index center 852 appeared in thesuperwide-angle image. However, since there are individual differenceand adjustment difference due to the user's physique etc. actually, thecenter 856 of the superwide-angle image does not match with thepositioning index center 852 usually.

It is preferable for a user to match the extraction center position to avisual field center of the user in a current posture or operation (i.e.,the positioning index center 852 in the superwide-angle image) ratherthan to match to the center 856 of the superwide-angle image shown bythe camera body 1.

Accordingly, a deviation amount of the positioning index center 852 fromthe center 856 of the superwide-angle image is measured, and theextraction center position is offset to a position based on thepositioning index center 852 that differs from the center 856 of thesuperwide-angle image. Moreover, the face direction that is detected bythe face direction detection unit 20 in that time is also offset in asimilar way.

Concrete offset methods will be described by referring to FIG. 22C andFIG. 22D. The deviation amount of the positioning index center 852 tothe center 856 of a superwide-angle image is measured. And the measureddeviation amount is divided into a lateral deviation amount 857 a and avertical deviation amount 857 b as shown in FIG. 22C. An offset amountis determined on the basis of the deviation amounts 857 a and 857 bafter performing a suitable conversion process corresponding to theprojection method of the entire field angle.

Moreover, as shown in FIG. 22D, the offset amount may be determinedafter applying the suitable conversion process to the superwide-angleimage corresponding to the projection method. That is, the deviationamount of the center 856 a from the positioning index center 852 in thesuperwide-angle image after conversion is measured. And the deviationamount is divided into a lateral deviation amount 857 c and a verticaldeviation amount 857 d. Then, the offset amount may be determined on thebasis of the deviation amounts 857 c and 857 d.

The offset method can be arbitrarily selected from among the methodsshown in FIG. 22C and FIG. 22D in consideration of the processing loadand the object of the camera system.

By performing the above-mentioned calibration operation for the frontdirection, a face direction of a user who wears the camera body 1, avisual field center in the face direction within a superwide-angleimage, and a face direction detected by the face direction detectionunit 20 are appropriately associated irrespective of individualdifference and adjustment difference.

The calibration operation for the front direction is described up tohere among the five directions (front, upper right, lower right, upperleft, and lower left). It is necessary to execute similar calibrationoperations for the remaining four directions.

Accordingly, when the process in the step S3110 in FIG. 21 is completed,the process proceeds to a step S3111. In the step S3111, when there is adirection for which the calibration operation is not performed among thefive directions, a target direction of the calibration operation ischanged, and the process returns to the step S3103. Thereby, thecalibration operation is similarly repeated for the remaining fourdirections other than the already finished front direction.

Although it is not shown in FIG. 21, when it is determined that there isno direction for which the calibration operation is not performed in thestep S3111, this process is finished as-is.

FIG. 23A through FIG. 23E are views for describing the calibrationoperation for an upper right direction of the user (the upper rightdirection in the superwide-angle image). FIG. 23A through FIG. 23Erespectively correspond to FIG. 22A through FIG. 22E and the fundamentaloperation is also identical. Accordingly, the common description isomitted.

As shown in FIG. 23A, the instruction display 855 displays a characterstring that instructs the user to locate the positioning index 851 atthe visual field center of the user in directing the face to the upperright.

FIG. 23B is a view showing a state where the user holds the calibrator850 to upper right according to an instruction shown by the instructiondisplay 855 in FIG. 23A. FIG. 23C is a schematic view showing the entiresuperwide-angle image that is caught by the image pickup lens 16 in thestate in FIG. 23B.

As shown in FIG. 23C, a deviation amount between the center 856 of thesuperwide-angle image and the positioning index center 852 is measuredfirst according to a concrete offset method. After that, the measureddeviation amount is divided into a radial deviation amount 857 e and anangular deviation amount 857 f. An offset amount is determined on thebasis of the deviation amounts 857 e and 857 f after performing asuitable conversion process corresponding to the projection method ofthe entire field angle.

Moreover, as shown in FIG. 23D, the offset amount may be determinedafter applying the suitable conversion process to the superwide-angleimage corresponding to the projection method. That is, the deviationamount of the center 856 a from the positioning index center 852 in thesuperwide-angle image after conversion is measured. And the deviationamount is divided into a radial deviation amount 857 g and an angulardeviation amount 857 h. Then, the offset amount may be determined on thebasis of the deviation amounts 857 g and 857 h.

The determination of the offset amount described using FIG. 22A throughFIG. 22E employs the method of dividing the deviation amount into thelateral deviation amount and vertical deviation amount. As compared withthis, the determination of the offset amount described using FIG. 23Athrough FIG. 23E employs the method of dividing the deviation amountinto the radial deviation amount and angular deviation amount. Thedifference in method is only for convenience of description, and eithermethod can be employed.

Moreover, the face direction detection unit 20 has obtained, as shown inFIG. 23 E, the throat position 206 and the chin position 207 ru requiredto calculate the face direction in directing the face to the upperright. Accordingly, the face direction of the user in looking in thedirection (in this case, the upper right direction) toward thepositioning index center 852 can be correctly measured irrespective ofindividual difference and adjustment difference of the user.

As mentioned above, the calibration operations for upper right, lowerright, upper left, and lower left directions in addition to the frontdirection will be performed in the calibration process shown in FIG. 21.Thereby, when the user turns the head in either of the upper, lower,right, and left directions, the face direction detection unit 20 cancorrectly detect the direction in which the user turns. Accordingly, theuser can use the camera body 1 appropriately irrespective of individualdifference and adjustment difference.

In the above description, the method of performing the calibrationoperation repeatedly for the five directions (front, upper right, lowerright, upper left, and lower left) is described to simplify thedescription.

However, the calibration operation is not limited to this method. Forexample, the following method may be employed. That is, a usercontinuously moves the calibrator 850 according to the instructiondisplay 855. At the same time, the user continuously catches thepositioning index 851 displayed on the calibrator 850 at the visualfield center. The user moves the calibrator 850 along a Z-shaped locus,a spiral locus, a polygonal locus, or the like. In this method, thedisplay-apparatus controller 801 transmits the calibration instructionsto the camera body 1 multiple times while the calibrator 850 is moving.

Whenever receiving the calibration instruction, the overall control CPU101 obtains the face direction detected by the face direction detectionunit 20 and the positional coordinate information about the positioningindex center 852 in the superwide-angle image picked up by the imagepickup unit 40, and saves them as history information. After that, theoverall control CPU 101 calculates the relationship of the extractioncenter position of the image and the face direction of the user bycombining the information extracted from the obtained historyinformation. Furthermore, in this method, the information extracted fromthe history information may be limited to the information obtained whenthe user looks at the positioning index 851. The information is limitedusing the information about the in-camera 805 and face sensor 806obtained by the calibrator 850 during movement of the calibrator 850.Thereby, the information obtained when the user is looking away is nolonger extracted from the history information, which raises the accuracyof calculation of the relationship.

Moreover, the display-apparatus controller 801 may transmit ameasurement value of the angular speed sensor 807 to the camera body 1together with the calibration instruction. In this case, the overallcontrol CPU 101 obtains movement information showing a moving locus ofthe calibrator 850 by the user and the position and posture of thecalibrator 850 from the transmitted measurement value of the angularspeed sensor 807. The movement information is also saved as the historyinformation. Thereby, the calibration operation can be performed easilyand correctly on the basis of the movement information based on themeasurement value of the angular speed sensor 807, the face directiondetected by the face direction detection unit 20, and the positionalcoordinate information about the positioning index center 852 in thesuperwide-angle image picked up by the image pickup unit 40.

In this case, the movement information based on the measurement value ofthe angular speed sensor 807 should be coincident with the movementinformation based on the positional coordinate information about thepositioning index 851. Accordingly, when the measurement value of theangular speed sensor 807 is used, it is required to synchronizecommunication between the camera body 1 and the calibrator 850.

As mentioned above, the second embodiment describes the calibrationmethod that enables to associate the face direction of the user with thecenter position of the target visual field 125 set in thesuperwide-angle image irrespective of individual difference andadjustment difference. In the meantime, the present disclosure is notlimited to the various configurations exemplified in the secondembodiment and various modifications are available within the scope ofthe present disclosure.

Next, a third embodiment will be described. In the third embodiment, amethod to prevent visually induced motion sickness caused by thesecondarily recorded image is described using FIG. 24A through FIG. 26.

This embodiment is described as a derivation from the first embodimentbasically. Accordingly, configurations of the camera system in the thirdembodiment that are identical to the configurations of the camera systemin the first embodiment are indicated by the same reference numerals andduplicated descriptions are omitted. A different configuration will bedescribed by adding details.

As a result of advance of imaging technology, a CG image just like aphotographed image and a powerful 3D video image can be easily enjoyed.

In the meantime, when such a 3D video image is an image with dynamicmovement like a VR video image or an image with much camera shake, thevisually induced motion sickness tends to occur while viewing the image.Since the visually induced motion sickness causes a symptom like motionsickness, interest in its safety measure is increasing.

The camera system shall be designed so that an image in a directiontoward which a user's face is directing is extracted and developed as-isin the recording area development process (the step S500). In such acase, when the user's face moves quickly during the image pickupoperation by the image pickup unit 40 (step S400), image scenes alsoswitch at a fast speed.

Although the user who moves the face quickly during the image pickupoperation by the image pickup unit 40 does not get sick, an appreciationperson who appreciates the image that is secondarily recorded in thestep S1000 may suffer from the visually induced motion sickness when theimage includes the image scene of the quick movement.

The above-mentioned publications disclose the technique of picking up animage in a direction toward which a face is directed, but they do notdisclose a counterplan to such visually induced motion sickness.Accordingly, this embodiment provides a camera system that prevents anappreciation person from suffering from the visually induced motionsickness. Therefore, even if the user moves the face quickly during theimage pickup operation by the image pickup unit 40, the camera systemcontrols so that the finished image does not include image scenes thatare switched at a fast speed.

As described using FIG. 8H through FIG. 8K or FIG. 10B through FIG. 10D,the user's face may turn in the upper/lower/right/left directions duringthe image pickup operation by the image pickup unit 40. Accordingly, thedirection and speed of the movement of the user's face are representedby an angular speed ω, and its moving amount is represented by an angleθ. The angular speed ω is calculated by dividing the angle θ detected bythe face direction detection unit 20 by a detection period.

Human actions that quickly move a face include looking back, a glance, amoving object observation, etc.

The looking back is an action that a person looks back when loud soundoccurs, for example. The glance is an action that a person once looks atsomething that caused worrisome change in a visual field and thenreturns the face to the former position because it is almostuninterested. The moving object observation is an action that a personcontinuously observes a moving object, such as a bird and a kite thatfly in the sky freely.

When such an action occurs during the image pickup operation by theimage pickup unit 40, and when an image in a direction toward which auser's face is directing is extracted and developed as-is in therecording area development process, an appreciation person whoappreciates a finished image may suffer from the visually induced motionsickness as mentioned above.

Accordingly, the overall control CPU 101 determines that the action thatquickly moves the face (one of the looking back, glance, and movingobject observation) occurs when the state where the angular speed ω isequal to or more than a threshold ω₀ is kept beyond a firstpredetermined time. Furthermore, when the overall control CPU 101determines that the occurred action is neither the glance nor the movingobject observation according to a method mentioned later using FIG. 25Aand FIG. 25B, the overall control CPU 101 determines that the action isthe looking back. In this case, the overall control CPU 101 does notimmediately extract the image in the direction toward which the user'sface is directing in the recording area development process. Instead,the overall control CPU 101 delays the extraction of the image withrespect to the movement of the user's face (delay extraction).

In this embodiment, the threshold ω₀ is set to n/8 rad/s. This is aspeed at which a face turns from the front (0°) to the just side (90°)in 4 seconds. In the meantime, the threshold ω₀ is not limited to n/8rad/s. For example, the threshold ω₀ may be set to (n·π)/x rad/s (x isany value) on the basis of a frame rate n fps.

The angular speed con can be calculated by the following formula 200 onthe basis of the angle θ_(n) and obtained time t_(n) of the image of thecurrent frame n and the angle θ_(n−1) and obtained time t_(n−1) of theimage of the previous frame n−1.

$\begin{matrix}{\omega_{n} = \frac{\theta_{n} - \theta_{n - 1}}{t_{n} - t_{n - 1}}} & {{Formula}200}\end{matrix}$

In the meantime, the angular speed ω may be an arithmetic mean ofangular speeds of x-frames from the angular speed ω_(n−x) of the framen−x to the angular speed ω_(n) of the current frame n.

Furthermore, although the predetermined period is set to 0.2 second inthis embodiment, it is not limited to this value.

Hereinafter, the delay extraction in a case where the user is lookingback will be described using FIG. 24A, FIG. 24B, and FIG. 24C.

Although the descriptions using FIG. 11A through FIG. 11F and FIG. 12Athrough FIG. 12G in the first embodiment take the distortion intoconsideration, the distortion of the image pickup lens 16 is not takeninto consideration in this embodiment to simply descriptions. Moreover,the following description assumes that the calibration process of thesecond embodiment has been applied to images of frames and a center ofan image of each frame is coincident with a visual field center of auser in picking up the image. Moreover, in order to describe a casewhere a face turns a just side, a case where light rays within themaximum field angle 192° are projected to the solid state image sensor42 is described as an example.

An area 4000 indicates the pixel area that can be picked up by the solidstate image sensor 42. An image 4001 (FIG. 24A and FIG. 24B) is an imageof the frame f_(n) that is extracted as the target visual field 125 inthe direction toward which the face is currently directing.

An image 4002 (FIG. 24A and FIG. 24B) is an image of the frame f_(n−1)that is extracted as the target visual field 125 in the direction towardwhich the face was directed at the previous time.

Hereinafter, a value d indicates a distance 4010 (FIG. 24A) from thecenter of the image 4002 of the frame f_(n−1) to the center of the image4001 of the frame f_(n).

An image 4003 (FIG. 24B) is extracted from the image projected to thearea 4000 as an image of a delay extraction frame f′_(n) in a case wherethe angular speed ω of the face based on the face direction detected bythe face direction detection unit 20 is equal to or more than thethreshold ω₀.

Hereinafter, a value d′ indicates a distance 4011 between the center ofthe image 4002 of the frame f_(n−1) and the center of the image 4003 ofthe delay extraction frame f′_(n).

A value d″ is a delay distance 4012 from the center of the image 4001 ofthe frame f_(n) to the center of the image 4003 of the frame f′_(n). Atthis time, the value d of the distance 4010 is larger than the value d′of the distance 4011 (d>d′).

Next, a method to determine the value d′ is described using FIG. 24C.Hereinafter, a case where a user quickly moves a face rightward from thefront (observation direction vo (vector information (0°, 0°))) to thejust side (90°) is described. In this case, an image 4021 of a framef_(n) extracted when the face is directed to the front (the observationdirection vo (vector information (0°, 0°))) is obtained first. After ashort period, an image 4022 of a frame f_(n+x) extracted when the faceis directed rightward to the just side (90°) is obtained.

In order to prevent the visually induced motion sickness, it shall benecessary to spend at least t seconds (for example, 4 seconds) forturning the face from the front to the just right side (90°), When theframe rate of the image is n fps (for example, 30 fps), the distance d′is obtained by the following equation.

$d^{\prime} = \frac{f_{n + x} - f_{n}}{n \cdot t}$

In the meantime, when the distance d″ from the frame f_(n) to the framef′_(n) becomes larger, the object at which the user looks may not bepicked up in the frame f_(n) because the face direction differs from therecording direction.

When the delay period becomes equal to or more than a predeterminedperiod Th_(delay) (second predetermined time), the delay extraction isstopped and extraction of a direction toward which the face is currentlydirecting (referred to as a current face direction) is started.

The delay period is a difference between start time t₀ at which thedelay starts (a step S4211 in FIG. 26) and current time t_(n) (a stepS4213 in FIG. 26) at which the face is continuously moving.

Although the predetermined value Th_(delay) is set as 1 second in thisembodiment, it is not limited to 1 second. For example, thepredetermined value Th_(delay) may be set to 20/n seconds based on theframe rate n fps. When the predetermined value Th_(delay) is 20/nseconds, the predetermined value Th_(delay) becomes shorter as the framerate becomes higher. Since the possibility of the visually inducedmotion sickness becomes lower as the frame rate becomes higher, theprocess can be returned to the extraction of the current face directionin a short delay period.

In the meantime, when the delay extraction is stopped and the extractionof the current face direction is restarted, the image scene switchessuddenly. Since such sudden switching of the image scene causesunnatural feeling to the user, image effects, such as fade-out andfade-in, may be employed.

Moreover, the locus of the current face direction is saved so that theextraction of the current face direction can be restarted. The savedlocus of the face direction in a case where it is determined that theuser is glancing is described using FIG. 25A.

The process that stops the delay extraction and restarts the extractionof the current face direction is executed when the delay period becomesequal to or more than the predetermined value Th_(delay) as mentionedabove. In addition, the process is also executed in a case of glance,i.e., the case where the user once changes the face direction to acertain direction and immediately returns to the previous direction.

FIG. 25A is the view showing the example of the locus of the facedirection in a case where the user is glancing. A center position 4101of an image of a frame f_(n−3) coincides with the user's visual fieldcenter in beginning of movement of the face. After that, the user'svisual field center sequentially moves to center positions 4102, 4103and 4104 of images of frames f_(n−2), f_(n−1), and f_(n). Hereinafter,such a movement of the user's visual field center is called a facemotion vector.

The user's visual field center stops at the center position 4104 for awhile, then moves to center positions 4105, 4106, and 4107 of images offrames f_(nx+1), f_(nx+2), and f_(nx+3), and stops at the centerposition 4107 of the image of the frame f_(nx+3). That is, the directionof the face motion vector from the position 4101 to the position 4104 isopposite to the direction of the face motion vector from the position4104 to the position 4107.

When frame groups of which motion vectors are mutually opposite aredetected as exemplified in FIG. 25A, the overall control CPU 101determines that the frame groups correspond to the glance.

In this case, the overall control CPU 101 performs the delay extractionfrom the position 4104 at which the face starts moving to the position4104 at which the motion vector starts moving conversely. This isbecause the position 4101 is considered as a position of an object thatthe user wants to glance.

In the meantime, after performing the delay extraction to the position4101, the overall control CPU 101 stops the delay extraction andrestarts the extraction of the current face direction to the position4107 at which the movement of the face stops.

Furthermore, when a body is detected near the center of the view fieldin the face direction during the movement of the user's face, and whenthe body keeps its position near the center of the view field in theface direction, the overall control CPU 101 determines that the user isobserving a moving object. In this case, the delay extraction is notperformed in this embodiment.

FIG. 25B is a view showing examples of images of frames of a case wherethe user is observing a moving object. A center position of an image4121 of a frame f_(n−3) coincides with the user's visual field center instarting of movement of the face. After that, the user's visual fieldcenter moves to center positions of images 4122, 4123, 4123, 4125, and4126 of frames f_(n), f_(n+1), f_(n+2), f_(n+3), and f_(n+4)

When detecting that the same object keeps its position near the centersof images of continuous frames as exemplified in FIG. 25B, the overallcontrol CPU 101 determines that the frames belong to a frame group ofthe moving object observation.

In this case, the overall control CPU 101 does not perform the delayextraction. This is because the delay extraction during the movingobject observation increases a possibility that an object is notcaptured in an image.

Moreover, when an appreciation person appreciates the video of theimages 4121 through 4126 extracted in response to the fast movement ofthe user's face during the moving object observation, the appreciationperson may suffer from the visually induced motion sickness.Accordingly, the overall control CPU 101 does not perform the imageextraction about the frame group of the moving object observation andrecords an image of the entire pixel area 4000 that can be captured bythe solid state image sensor 42.

It should be noted that margins called blind zones may be given to thethreshold ω₀, the predetermined period, and the predetermined valueTh_(delay).

Next, a visually-induced-motion-sickness prevention process according tothis embodiment will be described using a flowchart in FIG. 26. Itshould be noted that this process is executed whenever the image pickupunit 40 picks up a frame image in the step S400 during the video imagepickup operation.

In a step S4201, the overall control CPU 101 obtains the face direction(observation direction) recorded on the primary memory 103 in the facedirection detection process executed for the current frame image pickupoperation.

In a step S4202, the overall control CPU 101 obtains the position andsize (extraction area) of the image recording frame recorded on theprimary memory 103 in the recording-direction/area determination processexecuted for the current frame image pickup operation.

In a step S4203, the overall control CPU (a calculation unit) calculatesthe angular speed ω of the face on the basis of the face direction ofthe current frame image pickup operation obtained in the step S4201, theface direction of the previous frame image pickup operation stored inthe primary memory 103, and the frame rate. After that, the overallcontrol CPU 101 determines whether the face starts moving at the angularspeed ω beyond the threshold ω₀.

Specifically, when the user's face starts moving at the angular speed ωbeyond the threshold ω₀ beyond a predetermined period (0.2 seconds), theoverall control CPU 101 determines that the face starts moving at theangular speed ω beyond the threshold ω₀. When it is determined that theface starts moving (YES in the step S4203), the process proceeds to astep S4204. Otherwise (NO in the step S4203), the process returns to thestep S4201. That is, even if the user's face moves at the angular speedω beyond the threshold ω₀, when the period is less than thepredetermined period (less than a first predetermined period), theprocess returns to the step S4201. Moreover, when the angular speed ofthe face cannot be calculated in the step S4203 because the facedirection at the previous frame image pickup operation is not saved inthe primary memory 103, the process returns to the step S4201.

In the step S4204, the overall control CPU 101 determines whether theface moved more than the predetermined angle on the basis of the angularspeed ω of the face calculated in the step S4203. When it is determinedthat the face moved (YES in the step S4204), the process proceeds to astep S4206. Otherwise (NO in the step S4204), the process proceeds to astep S4205. It should be noted that the overall control CPU 101 maydetermine whether the face moved at the angular speed beyond thepredetermined angular speed beyond the predetermined period (0.2seconds) in the step S4204.

In the step S4205, the overall control CPU 101 determines whether themovement of the face stopped on the basis of the angular speed ω of theface calculated in the step S4203. When it is determined that themovement stopped (YES in the step S4205), the process returns to thestep S4201. Otherwise (NO in the step S4205), the process returns to thestep S4204.

In the step S4206, the overall control CPU 101 determines whether thepicked-up object is moving, i.e., determines whether the user isobserving a moving object. When it is determined that the object ismoving (YES in the step S4206), the process proceeds to a step S4207.Otherwise (NO in the step S4206), the process proceeds to a step S4208.

In the step S4207, the overall control CPU 101 determines not to performthe crop development process in the recording area development processof the current frame and to perform the development process ofentire-area RAW data obtained from the entire area of the solid stateimage sensor 42. Then, the process proceeds to the step S4205.

In the step S4208, the overall control CPU 101 stores the face directionat the current frame image pickup operation obtained in the step S4201to the primary memory 103. Then, the process proceeds to a step S4209.

In the step S4209, the overall control CPU (a delay unit) 101 determinesto perform the crop development process (to perform the delayextraction) in the recording area development process of the currentframe about the extraction area centering on the position shifted fromthe face direction of the previous frame by the distance d. After that,the process proceeds to a step S4210.

In the step S4210, the overall control CPU 101 determines whether thestart time t₀ of the time period stored in the primary memory 103 iscleared. When it is determined that the start time is cleared (YES inthe step S4210), the process proceeds to a step S4211. Otherwise (NO inthe step S4210), the process proceeds to a step S4212.

In the step S4211, the overall control CPU 101 stores current time asthe start time t₀ to the primary memory 103. Then, the process proceedsto the step S4212.

In the step S4212, the overall control CPU 101 determines whether themovement of the face stopped before the delay period reaches thepredetermined value Th_(delay) on the basis of the angular speed to ofthe face calculated in the step S4203. When it is determined that themovement stopped (NO in the step S4212), the process proceeds to a stepS4215. Otherwise (NO in the step S4212), the process proceeds to a stepS4213.

In the step S4213, the overall control CPU 101 stores current time astime t₀ to the primary memory 103. Then, the process proceeds to thestep S4214.

In the step S4214, the overall control CPU 101 calculates the delayperiod by subtracting the start time t₀ from the time t_(n) that arestored in the primary memory 103 and determines whether the delay periodis equal to or more than the predetermined period Th_(delay). When thedelay period is equal to or more than the predetermined periodTh_(delay) (YES in the step S4214), the process proceeds to the stepS4215. Otherwise (NO in the step S4214), the process returns to the stepS4206.

In the step S4215, the overall control CPU 101 clears the start time t₀stored in the primary memory 103. Then, the process proceeds to the stepS4216. In the step S4216, the overall control CPU 101 determines arecording direction and a field angle by therecording-direction/field-angle determination unit 30 on the basis ofthe face direction detected by the face direction detection unit 20.Then, the process proceeds to a step S4217.

In the step S4217, the overall control CPU 101 sets a flag to metadataof the current frame. Then the process returns to the step S4201. Theflag set to the metadata is used to determine timings at which imageeffects (fade effects), such as fade-in and fade-out mentioned above, inthe secondary recording process described in the step S1000 in the firstembodiment.

As mentioned above, in this embodiment, when the angular speed ω of theface becomes beyond the threshold ω₀, the frame in the face direction isnot extracted as-is and the frame is extracted according to the movementof the face. This has an effect to reduce the visually induced motionsickness.

Next, a fourth embodiment will be described. Fourth embodiment describeshow to correct the extraction area of an image depending on the movementspeed of the orientation a user's face using FIG. 27A, through FIG. 27F, FIG. 28A, and FIG. 28 B.

This embodiment is described as a derivation from the first embodimentbasically. Accordingly, configurations of the camera system in thefourth embodiment that are identical to the configurations of the camerasystem in the first embodiment are indicated by the same referencenumerals and duplicated descriptions are omitted. A differentconfiguration will be described by adding details.

A person's action to change an observation direction will be describedfirst. Usually, when a person finds an interested item in a peripheralarea of a visual field deviated from a center of the visual field andturns an observation direction toward the item, a face moves first and abody follows after movement of the face exceeds a certain amount.

That is, in such a case, the direction of the image pickup lens 16 ofthe image-pickup/detection unit 10 (FIG. 10A) in front of the claviclesdoes not change while only the face is changing the orientation in aninitial motion. After that, when the user starts changing an orientationof the entire body, the direction of the image pickup lens 16 of thecamera body 1 also moves. The following description presupposes such acharacteristic feature of a human action.

Moreover, when the face direction detection unit 20 detects a facedirection, variation due to a detection error occurs. When an extractionposition of an image is calculated on the basis of the detection resultof the face direction including the variation, blur like a result of acamera shake appears in a video image secondarily recorded in the stepS1000, which deteriorates the appearance. Accordingly, slight variationis removed by applying a low pass filter to the detection result of theface direction in order to correct a slight detection shake.

Moreover, if the face direction is detected following a momentarymovement (for example, a right and left check while walking along apublic road), a video image secondarily recorded in the step S1000 tendsto cause the visually induced motion sickness. Accordingly, in thisembodiment employs a process that removes (smooths) a slight movingcomponent of the face direction detected by following a momentarymovement for about 1 through 2 seconds. Thereby, the appearance of thevideo image secondarily recorded in the step S100 is improved.

Next, a summary of an extraction-area correction process in thisembodiment will be described using FIG. 27A through FIG. 27F.

A horizontal axis of each graph shown in FIG. 27A through FIG. 27Findicates elapsed time. A vertical axis in FIG. 27A indicates an angularmovement of an actual observation center. Vertical axes in FIG. 27B andFIG. 27C indicate an angle of a face direction. A vertical axis in FIG.27D indicate an angle of a direction of the image pickup lens 16. Andvertical axes in FIG. 27E and FIG. 27F indicate an angle of anextraction position. It should be noted that the upper direction in thevertical axis shows the right direction.

FIG. 27A is a graph showing movement of an actual observation center(face direction). The angle of the vertical axis of FIG. 27A indicates aface direction of a user with respect to a fixed location like a groundsurface (a ground standard) and does not indicate an angle showing aface direction detected by the face direction detection unit 20. Thatis, the graph in FIG. 27A shows that the user faces the front at thebeginning and starts turning the face rightward at about 1 second.

FIG. 27B is a graph showing the detection result (observation directionvi) of the face direction detection unit 20. The reason why the lineshowing the detection result in FIG. 27B is not smooth is because thedetection result contains variation due to a detection error asmentioned above. Accordingly, in this embodiment, a low pass filter isapplied to the detection result of the face direction detection unit 20.

Moreover, a process that removes (smooths) a quick change of the facedirection detected by following a momentary movement of the face is alsoperformed. FIG. 27B does not show such a quick change.

FIG. 27C is a graph showing a result of smoothing obtained by applyingthe low pass filter to the detection result of the face directiondetection unit 20 in FIG. 27B. As shown in FIG. 27C, the line showingthe detection result in FIG. 27B turns into the smooth line by applyingthe low pass filter. In the meantime, as a result of applying such afilter, the turning of the face from the front to the right is detectedat about 2 second in FIG. 27C. That is, delay (time lag) has occurred inthe graph in FIG. 27C from the graph in FIG. 27B that directlycorresponds to the movement in FIG. 27A. It should be noted that theangle of the vertical axes in FIG. 27B and FIG. 27C shows an angle fromthe direction of the image pickup lens 16 (the camera body 1 is madestandard) and is not the angle of the ground standard in FIG. 27A.

Moreover, in FIG. 27B, a tilt becomes gradual from about 4 second ascompared with FIG. 27A. This means that the moving speed of the facedirection detected by the face direction detection unit 20 is relativelyslowing down because the camera body 1 (the direction of the imagepickup lens 16) starts moving with the body of the user from about 4second as shown in FIG. 27D.

FIG. 27E shows a result of a simple addition method that calculates theextraction position (i.e., the observation direction as the center ofthe target visual field 125) by adding the moving amount of the camerabody (FIG. 27D) to the face direction detection result (FIG. 27C) thatis smoothed by applying the low pass filter. However, when theextraction position is calculated by this simple addition method, thecrop position does not follow the movement of the actual observationcenter. Accordingly, the video image finished in the secondary recordingprocess includes a scene where panning accelerates suddenly from about4.5 second from which the movement of the body starts.

That is, in order to eliminate discomfort to the movement of the actualobservation center, it is preferable to calculate the extractionposition (expectation value) so as to keep the panning approximatelyconstant as shown in FIG. 27F.

Accordingly, in this embodiment, the extraction position is calculatedso as to avoid the scene where panning accelerates suddenly as shown inFIG. 29E. When there are two moving speeds (0°/s and 10°/s) of theextraction position as shown in FIG. 27F, the expectation value shown inFIG. 27F is calculated by adding the moving amount of the camera body 1in FIG. 27D at timing preceding by the time lag (1 second in thisembodiment) to the face direction detection result in FIG. 27C.Actually, the moving speed of the extraction position is not limited tothe above two kinds and varies gradually. That is, the observationdirection is not accelerated suddenly and is not stopped suddenly.Slowdown is gradual. However, the expectation value cannot draw agradual slowdown curve according to the above-mentioned calculationmethod. Accordingly, in this embodiment, when the movement of the camerabody 1 stops, the moving speeds of the extraction position within aperiod from start to stop of the movement of the observation directionor within a past certain period are allocated among several frames sothat the expectation value will draw a gradual slowdown curve.

Hereinafter, the extraction-area correction process in this embodimentwill be described in order using flowcharts in FIG. 28A and FIG. 28B.Hereinafter, descriptions about the same portions as the first throughthird embodiments will be simplified or omitted.

FIG. 28A is the flowchart showing a subroutine of therecording-direction/area determination process in the step S300 in FIG.7A according to this embodiment.

In a step S4000 a, the observation direction vi obtained by the facedirection detection process in the step S200 is smoothed using the lowpass filter (a smoothing unit). As mentioned above using FIG. 27B, thisis because the observation direction vi has variation due to somedetection error. The low pass filter takes a simple moving average ofpast several times, for example, 5 to 10 times. At this time, the delayof tracking when the face direction is moved becomes larger as the timesof taking the average increases. Moreover, when the user turns the faceto the right and immediately turns to the left, the observationdirection vi in turning to the rightmost may not be detected.

Furthermore, since a mixture state of a detection error depends on adetection method, a degree of the smoothing may be changed according toa detection method. An application method of the low pass filter in thevertical direction may be changed from that in the lateral direction.

Moreover, a momentary movement of a face does not need to record in manycases from a viewpoint of storing user's experience as an image. Forexample, the user has no choice but to check safety of right and leftwhile walking as mentioned above. An image picked up at such a moment isnot needed to record. Accordingly, in this embodiment, the observationdirection vi obtained when a moved observation direction returns to theprevious direction within about 2 seconds is also smoothed in the stepS4000 a.

Although safety checks in the left-right direction and the lowerdirection are needed in many cases, there is little need of safety checkin the upper direction. Accordingly, the low pass filter may not beapplied to the upward movement.

When the extraction area is determined by the process in the steps S301through S304 (FIG. 7D), the overall control CPU (a second calibrationunit) 101 proceeds with the process to a step S4000 and executes theextraction-area correction process.

After that, the extraction area after the correction is recorded in thestep S305, and the process exits this subroutine. The extraction-areacorrection process is described using the flowchart in FIG. 28B.

FIG. 28B is the flowchart showing the extraction-area correction processin the step S4000. In a step S4001 in FIG. 28B, the overall control CPU(moving speed calculation unit) 101 obtains gyro information, i.e.,movement (a gyro moving amount) of the camera body 1 in the currentframe, from the angular speed sensor 107.

Although the angular speed sensor 107 is used in this embodiment,another sensor may be used as long as the movement of the camera body 1can be detected. For example, a magnetometric sensor (not shown) thatmeasures a size and direction of a magnetic field may be used, and theacceleration sensor 108 that detects acceleration may be used.Furthermore, a method that extracts a feature point, detects a motionvector by calculating a moving amount of the feature point, andcalculates the moving amount of the camera body 1 may be used. A featurepoint can be extracted by a known method. For example, a moving amountcan be calculated by calculating a position at which difference becomessmall by subtracting a plurality of edge images in a deviated state thatare extracted by applying a bandpass filter to images obtained byextracting only luminance information from two image. Although thismethod increases a calculation amount, since the hardware like theangular speed sensor 107 becomes unnecessary and the weight saving ofthe camera body 1 is available, it is one of the preferable aspects.

The following description exemplifies a case where the gyro informationis obtained from the angular speed sensor 107. In a step S4002, a movingspeed (gyro moving speed) of the camera body 1 is calculated from thegyro information obtained in the step S4001 and the gyro informationobtained in the past.

In a step S4003, it is determined whether the gyro moving speedcalculated in the step S4002 is slowing down. When the moving speed isnot slowing down (NO in the step S4003), the process proceeds to a stepS4004. Otherwise (YES in the step S4003), the process proceeds to a stepS4006.

In the step S4004, the overall control CPU (the second calibration unitand observation direction correction unit) 101 calculates the movingspeed of the extraction position (an extraction-position moving speed)from the extraction position determined in the step S304 and theextraction position obtained in the past. Next, the overall control CPU101 obtains a subtraction amount by subtracting the gyro moving speedobtained at timing preceding by the time lag caused by applying the lowpass filter from the calculated extraction-position moving speed.

In a step S4005, the overall control CPU 101 stores theextraction-position moving speed and subtraction amount that areobtained in the step S4004 to the primary memory 103. And then, theprocess exits this subroutine.

In the step S4006, the overall control CPU 101 calculates theexpectation value by allocating the sum total of the subtraction amountsstored in the primary memory 103 among the extraction-position movingspeeds stored in the primary memory 103 so that the variation of theextraction-position moving speed in a past certain period will becomeconstant. And then, the process exits this subroutine. The past certainperiod may be a period from the start of actual movement of theextraction position to the present, or may be a period from detection ofmovement of the camera body 1 by the angular speed sensor 107 to thepresent. Moreover, in order to simplify the process, the past certainperiod may be a fixed period of 0.5 through 3 seconds. It should benoted that an expectation value prior to the past certain period is setto the extraction-position moving speed obtained in the step S4004.

The following Table 1 shows variations of the data (speeds) of thegraphs shown in FIG. 27A through FIG. 27F. That is, theextraction-position moving speeds determined in the step S304 are shownin a line C of Table 1. The gyro moving speed calculated in the stepS4002 are shown in a line D of Table 1. Moreover, the expectation valuescalculated in the step S4006 is shown in a line E of Table 1.

TABLE 1 0-1 sec 1-2 sec 2-3 sec 3-4 sec 4-5 sec 5-6 sec (A) Movement of0°/s 10°/s  10°/s 10°/s 10°/s 10°/s Observation center (B) Detected Face0°/s 10°/s  10°/s 10°/s  0°/s  0°/s Direction (C) Detected Face 0°/s0°/s 10°/s 10°/s 10°/s  0°/s Direction after Smoothing (D) Moving Amount0°/s 0°/s  0°/s  0°/s 10°/s 10°/s of Camera (E) Extraction 0°/s 0°/s10°/s 10°/s 20°/s 10°/s Position (Simple Addition) (F) Expectation 0°/s0°/s 10°/s 10°/s 10°/s 10°/s Value (Fourth Embodiment)

Hereinafter, the subroutine of the extraction-area correction process inFIG. 28B will be described about a case where the user first faces thefront and gradually turns the face to the right as shown in Table 1.

Since the user looks at the front at the beginning, the gyro movingspeed calculated in the step S4002 becomes about 0°/s. That is, it isdetermined that the gyro moving speed is not slowing down in the stepS4003, and the process proceeds to the step S4004. In this case, sincethe position of the face does not change, the extraction-position movingspeed also becomes 0°/s. Moreover, the subtraction amount calculated inthe step S4004 also becomes 0°/s.

Although the user starts turning the face to the right at about 1second, the extraction-position moving speed still keeps 0°/s. becauseof the time lag due to the low pass filter as shown in FIG. 27C. In themeantime, since the camera body 1 does not move, the gyro moving speedis about 0°/s as shown in FIG. 27D. Accordingly, the subtraction amountcalculated in the step S4004 also becomes 0°/s like the time when theuser still faces the front.

When the user further turns the face to the right at about 2 second, theextraction-position moving speed becomes 10°/s as shown in FIG. 27C. Inthe meantime, since the camera body 1 does not move, the gyro movingspeed is about 0°/s as shown in FIG. 27D. Accordingly, the subtractionamount calculated in the step S4004 becomes 10°/s.

When the user further turns the face to the right at about 4 second, theuser's body starts turning to the right. That is, since the direction ofthe camera body 1 changes as shown in FIG. 27D, the gyro moving speedbecomes 10°/s. Since the user's body starts turning, the actual angularspeed of the face slows down by a relative speed between the camera body1 and face direction as shown in FIG. 27B. In the meantime, theextraction-position moving speed shown in FIG. 27C still keeps 10°/sbecause of the time lag due to the low pass filter. Accordingly, thesubtraction amount calculated in the step S4004 becomes 10°/s by takingthe time lag into consideration,

When the user further turns the face to the right at about 5 second, thegyro moving speed still keeps 10°/s (FIG. 27D). In the meantime, theextraction-position moving speed shown in FIG. 27C slows down andbecomes 0°/s Accordingly, the subtraction amount calculated in the stepS4004 becomes −10°/s.

When the user finishes the action turning to the right after 6 second(not shown in FIG. 27A through FIG. 27F), the gyro moving speed becomes0°/s and the process is allowed to proceed to the step S4006 in thiscase. In this case, the sum total of the subtraction amounts calculatedup to now and stored in the primary memory 103 becomes +10°/s. Theexpectation value is calculated by allocating the sum total of thesubtraction amounts so that the variation of the extraction-positionmoving speeds stored in the primary memory 103 in the past certainperiod will become constant. In this case, the extraction-positionmoving speeds shown in FIG. 27C in the period from start of accelerationup to now (2 second through 6 second) are 10°/s, 10°/s, 10°/s, and 0°/sas shown in Table 1. Accordingly, all the expectation values in theperiod from 2 second to 6 second are set to 10°/s so as to keep thevariation of the extraction-position moving speed in this periodconstant (no variation in this embodiment).

Although the data are described by every second in this embodiment inorder to simplify the description, the frame rate of the video imagepickup operation is usually 24 through 60 fps. In the meantime, since itis not necessary to detect the face direction and gyro moving speed at60 times per second in many cases, the timing at which the facedirection detection process and extraction-area correction process areexecuted is preferably changed from the image pickup timing. Forexample, even when the frame rate of the video image pickup operation is60 fps, the timing at which the face direction detection process andextraction-area correction process may be set to 10 fps. The timing canbe changed suitably in consideration of a usage, power consumption, etc.

As described above, this embodiment shows the example that keeps themoving speed of the observation direction constant so as to avoid badappearance of the video image due to change of the moving speed of thevisual field caused when the movement of the face and the movement ofthe user's body (the camera body) are combined during great movement ofthe observation direction.

Although this embodiment shows the example that crops thesuperwide-angle image according to the observation direction. thepresent disclosure is not limited to this. For example, the overallcontrol CPU (an image pickup direction changing unit) 101 may change animage pickup direction of the image pickup unit 40 according to theobservation direction. In this case, the camera body 1 is required toprovide a mechanism (drive mechanism) that mechanically changes theimage pickup direction of the image pickup unit 40, specifically thedirection of the image pickup lens 16 and solid state image sensor 42,in a yaw direction and a pitch direction.

Moreover, the process that smooths the face direction detection resultshown in this embodiment is preferably performed when the overallcontrol CPU (the image stabilization unit) 101 performs the imagestabilization process described in the first embodiment, because theimage stabilization process causes the delay of tracking of the facedirection.

Next, a fifth embodiment will be described. The fifth embodimentdescribes a method for reducing difference between a user's visual fieldand secondarily recorded image (hereinafter referred to as a “recordedimage”) caused by parallax between an eye position of a user and a wornposition of the image-pickup/detection unit 10 using FIG. 29A throughFIG. 34C.

This embodiment is described as a derivation from the first embodimentbasically. Accordingly, configurations of the camera system in the fifthembodiment that are identical to the configurations of the camera systemin the first embodiment are indicated by the same reference numerals andduplicated descriptions are omitted. A different configuration will bedescribed by adding details. In order to support understanding, thedifference between the user's visual field and recorded image will bedescribed first.

FIG. 29A and FIG. 29B are schematic views for describing a relationshipbetween the visual field of the user 5010 and target visual field in acase where a short distance object is an observation target 5020 in thefirst embodiment.

FIG. 29A is the schematic view showing an image 5900 including theobservation object 5020 captured by the solid state image sensor 42.FIG. 29B is the schematic view showing a positional relationship betweenthe user 5010 and the observation object 5020.

As shown in FIG. 29B, when the observation object 5020 is below a heightof a user's eyes 5011, a face direction 5015 of the user 5010 turnsdownward. At this time, the observation object 5020 with a backgroundlike a floor (not shown) is caught in the visual field of the user 5010.

In the first embodiment, the observation direction 5040 (FIG. 29B)parallel to the user's face direction 5015 detected by the facedirection detection unit 20 is set as a recording direction.Accordingly, when the short distance object is the observation object5020 as shown in FIG. 29B, an area 5045 that does not include theobservation object 5020 will be set as the target visual field.

In such a case, even if a background (for example, a ceiling (notshown)) caught by the image-pickup/detection unit 10 will be differentfrom the background (for example, a floor (not shown)) of the visualfield of the user 5010, the observation direction should be set to adirection 5030 so that an area 5035 including the observation target5020 will be a target visual field.

The above issue is caused by the parallax due to the difference betweenthe position of the eye 5011 of the user 5010 and the worn position ofthe image-pickup/detection unit 10. Accordingly, in this embodiment, aparallax correction mode process that appropriately adjusts therecording direction set on the basis of the face direction of the user5010 corresponding to the parallax is executed.

FIG. 31 is a block diagram showing a hardware configuration of thecamera body 1 according to this embodiment. The hardware configurationof the camera body 1 in this embodiment differs from that of the camerabody 1 in the first embodiment shown in FIG. 5 in that a distance sensor5100 is added. In this embodiment, the distance sensor 5100 is providedin an outer edge of the stop switch 15 as shown in FIG. 30. However, amount position of the distance sensor 5100 is not limited to a certainposition.

The distance sensor 5100 measures a distance to an object. It should benoted that the configuration of the distance sensor 5100 is not limitedin particular. In this example, the distance sensor 5100 is an activetype sensor that projects infrared light, laser, millimeter wave, etc.to an object and measures a distance to the object by receiving itsreflection. Moreover, the distance sensor 5100 may be a passive typesensor that measures a distance to an object on the basis of phasedifference of incident light through the image pickup lens 16. Thedistance sensor 5100 is connected to the overall control CPU 101 and iscontrolled by the overall control CPU 101.

FIG. 32A and FIG. 32B are schematic views for describing a relationshipbetween a user, a calibrator 850, and a target visual field 5080 duringa calibration process including the parallax correction mode process inthis embodiment. FIG. 32A is the schematic view showing an image 5900including the calibrator 850 captured by the solid state image sensor42. FIG. 32B is the schematic view showing a positional relationshipbetween the user 5010 and the calibrator 850.

The target visual field 5080 in FIG. 32A is a target visual field in acase where the calibration process including the below-mentionedparallax correction mode process has not yet applied and a facedirection 5015 detected by the face direction detection unit 20 isdirected to the front.

In the meantime, a target visual field 5090 in FIG. 32A is a targetvisual field in a case where the calibration process including thebelow-mentioned parallax correction mode process has already applied andthe face direction 5015 detected by the face direction detection unit 20is directed to the front.

FIG. 33A is a flowchart showing the parallax correction mode processthat is a part of the preparation process in the step S100 in FIG. 7Aaccording to this embodiment. Hereinafter, details of this process willbe described also by using FIG. 32A and FIG. 32B.

In the preparation process in the step S100 in FIG. 7A, when theparallax correction mode starts by an operation of the user 5010 to thecalibrator 850 (a step S5101), the display-apparatus controller 801displays the positioning index 851 (a step S5102).

Subsequently, the display-apparatus controller 801 designates a position(designation position) to which the user should hold the calibrator 850.Specifically, the display-apparatus controller 801 instructs the user5010 to locate the positioning index 851 to the front at height of agaze by giving an instruction display similar to the instruction display855 shown in FIG. 22A (a step S5103).

After checking the instruction display, the user 5010 holds thecalibrator 850 at the designation position designated in the step S5103and directs the face direction 5015 toward the positioning index 851(the front). At this time, the user 5010, positioning index 851, andimage-pickup/detection unit 10 constitute the positional relationshipshown in FIG. 32B.

After that, when determining that the user looked at the positioningindex center 852 in the visual field center, the display-apparatuscontroller 801 measures a distance 5050 (FIG. 32B) between theimage-pickup/detection unit 10 and the positioning index 851 with thedistance measurement sensor 5100 (a step S5104).

Subsequently, the overall control CPU 101 detects a horizontal axis 5060of the image-pickup/detection unit 10 by the angular speed sensor (aposture detection unit) 107 (a step S5105). Thereby, a horizontalposition 5065 of the image 5900 (FIG. 32A) captured by the solid stateimage sensor 42 is specified.

Moreover, the overall control CPU 101 obtains a distance 5855 (FIG. 32A)between the center of the positioning index 851 and the horizontalposition 5065 on the image 5900 in the step S5105. After that, theoverall control CPU (an angle calculation unit) 101 calculates an angle5055 (FIG. 32B) between the horizontal axis 5060 and the direction ofthe positioning index 851 seen from the image-pickup/detection unit 10.This calculation is performed using the distance 5855 and theinformation about a relation between a point on the image 5900 and anincident angle of a light ray that images on the point. The informationis saved in a memory (for example, the internal nonvolatile memory 102).

After that, a step S5106, the overall control CPU (a vertical distancecalculation unit) 101 calculates a vertical distance 5070 between theimage-pickup/detection unit 10 and the eye 5011 of the user 5010 usingthe distance 5050 and the angle 5055 calculated in the step S5105. Then,the process exits this subroutine.

In this embodiment, the vertical distance 5070 between theimage-pickup/detection unit 10 and the eye 5011 of the user 5010 ismeasured by the method different from that of the second embodiment.However, the measurement method is not limited to this. For example, thevertical distance 5070 between the image-pickup/detection unit 10 andthe eye 5011 of the user 5010 may be measured by the method described inthe second embodiment, or the user 5010 may input the value of thevertical distance 5070 directly.

Since the calibration process including the parallax correction modeprocess in this embodiment is basically identical to the process in thesteps S3101 through S3111 in FIG. 21 executed in the second embodiment,its description is omitted.

However, in the step S3110, a process to correct the parallax based onthe vertical distance 5070 (FIG. 32B) calculated by the parallaxcorrection mode process in FIG. 33A is added to the process described inthe second embodiment. That is, the calibration such that the visualfield of the user 5010 matches the target visual field 125 in infinityis performed.

FIG. 33B is a flowchart showing a subroutine of arecording-direction/area determination process in the step S300 in FIG.7A according to this embodiment. Hereinafter, this process will bedescribed by also referring to FIG. 34A, FIG. 34B, and FIG. 34C. Thesteps in FIG. 33B that are identical to that in FIG. 7D are indicated bythe same reference numerals and duplicated descriptions are omitted.

In FIG. 33B, the overall control CPU 101 first obtains distanceinformation about an available image pickup area (image pickup targetarea) with the distance sensor (distance measurement unit) 5011 (a stepS5301).

In the next step S5302, the overall control CPU (a creation unit) 101creates a defocus map 5950 (FIG. 34A; distance map information) on thebasis of the distance information (a measurement result by the distancesensor 5100) obtained in the step S5301.

The defocus map 5950 in FIG. 34A is created when the situation shown inFIG. 34C where the observation object 5020 is appearing indoors ispicked up. In order to show the distance information in the defocus map5950 intelligibly, six distance areas A1 through A6 that are divided bythe distance from the image-pickup/detection unit 10 are indicatedstepwise. The distance area A1 is the nearest to theimage-pickup/detection unit 10. In the meantime, the defocus map may becreated stepless.

In the next step S5303, the overall control CPU 101 calculates thedirection of the observation object 5020 seen from theimage-pickup/detection unit 10 on the basis of the defocus map 5950,face direction 5015, and vertical distance 5070 (FIG. 32B). That is, theparallax correction is applied to the observation direction establishedon the basis of the face direction.

After that, the process in the steps S301 through S305 in FIG. 7D isperformed, and the process exits this subroutine.

Use of the defocus map 5950 created in this way and the detection resultof the face direction 5015 enables the calculation of the direction ofthe observation object 5020 seen from the image-pickup/detection unit10. Since there is the parallax described using FIG. 29A and FIG. 29B,the distance to the observation object 5020 cannot be measured with thedistance sensor 5100 unless creating the defocus map 5950.

The degree of the influence of the parallax described in this embodimentdepends on the distance between the user 5010 and the observation object5020. When an observation object is distant from the user 5010, theinfluence of the parallax can be disregarded. In such a case, the imagecan be extracted by the target visual field including the observationobject and can be recorded by the recording-direction/area determinationprocess in the first embodiment. For example, when the user 5010observes an observation object 5021 (FIG. 34C) that is positioned in themiddle distance area A5 distant to some extent from the user 5010, theparallax correction may not be applied to the recording direction in thestep S5303. This is because the observation object 5021 is also includedin the target visual field 5043 (FIG. 34B) established according to therecording direction (observation direction) 5041 that is estimated onthe basis of the face direction 5016 detected by the face directiondetection unit 20.

In the meantime, this embodiment can extend an allowable range of thedistance between the user 5010 and the observation object of the user5010 in which the observation object is held within the target visualfield to the nearer side than the first embodiment. For example, theuser 5010 shall be observing the observation object 5020 (FIG. 34A) thatis positioned in the nearest area A1 of which the distance from the user5010 is short. In this case, the observation direction (recordingdirection) 5040 is estimated on the basis of the face direction 5015detected by the face direction detection unit 20 in the firstembodiment. However, the target visual field 5042 (FIG. 34B) establishedaccording to this observation direction 5040 does not include theobservation object 5020. In the meantime, in this embodiment, theparallax correction is applied to the observation direction 5040 in thestep S5303 in FIG. 33B. As a result, the target visual field 5036including the observation object 5020 is established according to theparallax-corrected recording direction. Accordingly, an observationobject of which the distance to the user 5010 is short to such an extentthat influence of parallax cannot be disregarded, such as theobservation object 5020, can be also satisfactorily picked up.

Moreover, according to this embodiment, an observation object positionedin the middle distance area A5 can be recorded at nearer position to thecenter of the target visual field. For example, when the user 5010 isobserving the observation object 5021 (FIG. 34A) positioned in themiddle distance area A5, if the parallax correction is not applied tothe recording direction 5041 like the first embodiment, the targetvisual field 5043 in which the observation object 5021 is located in anupper end will be established. In the meantime, in this embodiment,since the parallax correction is applied to the recording direction 5041in the step S5303 in FIG. 33B, a recording area (target visual field)5037 in which the observation object 5021 is located at the center isgenerated according to the parallax-corrected recording direction.

In this way, when the parallax correction of this embodiment is applied,an observation object can be captured at nearer position to a center ofan extracted image in comparison with the first embodiment.

In this embodiment, the parallax correction is performed in thecalibration so that a visual field of a user matches a target visualfield in infinity. And then, when an image is picked up, the parallaxcorrection is applied so that deviation of recording directions beforeand after the correction becomes larger as a distance between a user andan observation object becomes shorter. In the meantime, the parallaxcorrection of this embodiment may be applied in the calibration processin the second embodiment to an object that is closer to the user thanthe position of the calibrator 850 or an object that is more distantfrom the user than the position of the calibrator 850.

Next, a sixth embodiment will be described. In the sixth embodiment, anextraction-area determination method used when calculation of anobservation direction fails will be described using FIG. 35, FIG. 36A,and FIG. 36B.

This embodiment is described as a derivation from the first embodimentbasically. Accordingly, configurations of the camera system in the sixthembodiment that are identical to the configurations of the camera systemin the first embodiment are indicated by the same reference numerals andduplicated descriptions are omitted. A different configuration will bedescribed by adding details.

In the first embodiment, as shown in FIG. 7A, the target visual field isestablished in the recording-direction/area determination process in thestep S300 on the basis of the observation direction calculated from theface direction detected by the face direction detection unit 20 in thestep S200. However, the face direction detection unit 20 may be coveredby obstacles, such as a collar and hair, may break down, or may separatefrom the user. In such a case, the face direction of the user cannot beobtained and an image of a target visual field that the user wanted topick up cannot be picked up.

In JP 2007-74033A, when the second camera that is used to capture a usercannot detect a user, detection of a user is retried without storingfailure of detection in a history of detection information about a user.Moreover, if detection of the face direction fails during an imagepickup operation by tracking a face direction, an image that does notlargely depart from user's intention is picked up by determining animage pickup direction depending on a situation.

Against this, in this embodiment, when a face direction of a user can bedetected, the face direction is detected by the face direction detectionunit 20 and picks up an image of a target visual field according to arecording direction that is calculated on the basis of the observationdirection as with the first embodiment. In the meantime, when a facedirection of a user cannot be detected and an observation direction ofthe user cannot be calculated, an image of a target visual field towhich user's intention is reflected is picked up. That is, in thisembodiment, after the face direction detection process in the step S200is completed, an observation direction determination process is executedbefore executing the recording-direction/area determination process inthe step S300. In the observation direction determination process, whenthe face direction detection unit 20 fails in detection of a user's facedirection, an observation direction is estimated by determining user'sintention according to a situation. That is, an image of a target visualfield in a recording direction based on a factor other than theobservation direction calculated from a face direction is picked up.

FIG. 35 is a flowchart of the observation direction determinationprocess according to this embodiment executed by the overall control CPU101. Hereinafter, this process will be described by also using FIG. 36Aand FIG. 36B.

In a step S6001, it is determined whether a face direction is detectedby the face direction detection unit 20. When the face direction isobtained, the process proceeds to a step S6004. In the step S6004, theoverall control CPU (a mode switching unit) 101 switches a mode of thisprocess to a face direction mode (first image pickup mode) and decidesthe observation direction calculated from the face direction by themethod shown in the first embodiment as the recording direction. Afterthat, the process exits this subroutine.

In the meantime, when a face direction is not obtained (NO in the stepS6001), the overall control CPU (mode switching unit) 101 proceeds withthe process to a step S6002 in order to shift to another mode. In thestep S6002, it is determined whether there is any object that wastracked in the past.

The determination process in the step S6002 will be described using FIG.36A that shows relationship between an observation direction detectionstate of the user and a pickup image for every frame.

In FIG. 36A, “n” denotes a frame number of the image, “0” denotes ahorizontal moving angle of a user's face, a user's state showspositional relationship between the user and an observation object ineach frame. Moreover, an entire image shows a superwide-angle imagepicked up by the image pickup unit 40 in each frame, and a pickup imageshows an image that is secondarily recorded in each frame andcorresponds to an area shown by a dotted line in the entire image.

As shown in each screen of the user's state in FIG. 36A, the user isobserving an object shown by a quadrangle in a bottom position of thescreen as an observation object. FIG. 36A exemplifies a case where auser's observation direction cannot be detected in the fifth frame(n=5).

In this embodiment, a period including four previous frames based on acurrent frame is defined as a predetermined period. When an object thatcan be determined as the same in three or more times within thepredetermined period is included in the pickup image, it is determinedthat there is an object that was tracked in the past.

As shown in FIG. 36A, although the moving angle θ varies by every +10°from the first frame (n=1) to the fourth frame (n=4), the object shownby the quadrangle that can be determined as the identical object isincluded in the pickup image. Accordingly, in the fifth frame (n=5), itis determined that there is an object that was tracked in the past. Itshould be noted that the criterion of the determination in the stepS6002 may be changed corresponding to the detection cycle of the facedirection or the accuracy of the face direction detection unit 20.

Returning back to FIG. 35, when it is determined that there is theidentical object that was tracked in the past predetermined period (YESin the step S6002), the process proceeds to the step S6005.

In the step S6005, the mode of this process is switched to a past-objecttracking mode (second image pickup mode) in which a past-objectdirection is determined as the recording direction, and the recordingdirection is determined so as to track the past object. And then, theprocess proceeds to a step S6008. In this way, in this embodiment, evenif the face direction cannot be detected, since the mode is switched tothe past-object tracking mode and the recording direction is determinedwhen there is an object that was tracked in the past, the user'simmediately preceding intention can be reflected to the image. Since amethod of recognizing an object within a pickup image and an objecttracking detection method performed by the overall control CPU (anobject recognition unit) 101 are publicly known, their detaileddescriptions are omitted.

In the meantime, when it is determined that there is no object that wastracked in the past (NO in the step S6002), the process proceeds to astep S6003. In the step S6003, it is determined whether the objectregistered into the internal nonvolatile memory (an object registrationunit) beforehand is detected in the newest pickup image.

In this embodiment, a user designates an image in which a person thatthe user wants to pick up from among images stored in the displayapparatus 800. The display-apparatus controller 801 recognizes featuresof the person and registers the object beforehand by transmitting thefeatures to the overall control CPU 101 in the camera body 1. It shouldbe noted that an object detected in the step S6003 is not limited tothis. For example, an object included in a pickup image obtained at areading completion timing or other detection timings may be detected inthe step S6003. Moreover, whether the object registered beforehandmatches the object in the newest pickup image is determined with apattern matching technique. Since the pattern matching technique ispublicly known, its detailed description is omitted.

When it is determined that the object registered beforehand is detectedin the newest pickup image (YES in the step S6003), the process proceedsto a step S6006. In the step S6006, the mode of this process is switchedto a registered-object tracking mode (third image pickup mode) in whicha registered-object direction is determined as the recording direction,and the recording direction is determined so as to track the registeredobject. And then, the process proceeds to the step S6008.

In the meantime, when it is determined that the object registeredbeforehand is not detected in the newest pickup image (NO in the stepS6003), it is determined that an observation object cannot be estimated,and the process proceeds to the step S6007.

In the step S6007, the overall control CPU (a field-angle change unit)101 switches the mode of this process to an object lost mode (a fourthimage pickup mode) in which the recording direction prior to the failureof the face direction detection is kept and the image pickup field angleis widened than a prescribed field angle. After that, the processproceeds to the step S6008. It should be noted that the recordingdirection in the object lost mode may be continuously moved by thechange amount of the observation direction prior to the failure of theface direction detection.

Hereinafter, a case proceeds to the step S6007 entering into the objectlost mode is described using FIG. 36B. FIG. 36B exemplifies a case wherethe user's observation direction cannot be detected in the fifth frame(n=5).

In the example in FIG. 36B, a main object is not found from the firstframe (n=1) to the fourth frame (n=4) and an object registeredbeforehand is not found in the pickup image of the fifth frame (n=5).Accordingly, the observation direction in the fifth frame (n=5) is movedrightward in the entire image by inertia of the movement in the firstthrough fourth frames. Moreover, the field angle extracted from theentire image is changed to a wider angle.

In the step S6008, when the recording direction is determined on thebasis of a factor other than the face direction in either of the stepsS6005 through S6007, the overall control CPU (a notification unit) 101notifies the user of an error (a detection error) showing that the facedirection detection failed. After that, the process exits thissubroutine. In this embodiment, a warning is output to the user usingthe vibrator 106 in FIG. 5. The notification method in the step S6008 isnot limited to this. Other notification methods, such as a warning usingthe LED 17, and a display of a warning message on a terminal like thedisplay apparatus 800 that cooperates with the camera body 1, may beemployed.

As mentioned above, in this embodiment, since the recording directionand the field angle are changed according to a situation when the facedirection cannot be detected, the user can avoid a pickup miss of theimage of the target visual field that the user inherently wants to pickup.

That is, in this embodiment, when a face direction cannot be detectedand when the object that was tracked in the past or the object that isregistered beforehand is detected, the object is tracked. In themeantime, when such an object cannot be detected, the image pickup fieldangle is widened than the prescribed field angle in order to avoid thepickup miss and to facilitate re-detection of an object.

Thereby, a situation where an image that the user does not intend ispicked up because of failure of face direction detection can beprevented.

Although the process in the steps S6001 through S6008 is performed forevery frame, the mode can be changed according to a mode determinationinformation, such as information about whether the face direction isobtained from the face direction detection unit 20, even after shiftingto each mode. For example, in this embodiment, when the object that isregistered beforehand is detected as a result of widening the fieldangle in the object lost mode, the mode is shifted to theregistered-object tracking mode in which the direction of the detectedobject is determined as the recording direction. In this case, thewidened field angle is restored to the prescribed field angle.

Moreover, although the mode is changed by one-time determination in thisembodiment, the mode may be shifted on the basis of multiple-timedeterminations according to the frame rate or the performance in theface direction detection.

Next, a seventh embodiment will be described. In the seventh embodiment,a method to determine an observation direction according to an accuracy(reliability) of face direction detection will be described using FIG.37A through FIG. 40.

This embodiment is described as a derivation from the first embodimentbasically. Accordingly, configurations of the camera system in theseventh embodiment that are identical to the configurations of thecamera system in the first embodiment are indicated by the samereference numerals and duplicated descriptions are omitted. A differentconfiguration will be described by adding details.

The sixth embodiment prevents an image pickup operation in a recordingdirection that the user does not intend by switching the mode thatdetermines the observation direction according to whether the facedirection can be detected. In the meantime, when a user's face directioncannot be stably detected like JP 2007-74033A, an image may be pick upat a field angle that a user does not intend. When theimage-pickup/detection unit 10 of the camera body 1 is worn in front ofthe clavicles as shown in FIG. 1B, the detection accuracy of the facedirection detection may fall under the influence of a collar, hair, etc.If the detection accuracy falls, the face direction cannot be stablydetected.

When the user turns the face in the lateral direction (FIG. 37B and FIG.37C), an area where a jaw and a cheek are hidden by a body or a shoulderbecomes larger than that in a case where the user faces the frontdirection (FIG. 37A). That is, the camera body 1 has such acharacteristic that the face area that can be used to detect the facedirection becomes narrow in some face directions. The possibility ofdrop of the detection accuracy increases in such face directions. Thischaracteristic greatly depends on the wearing point of the camera body 1selected by the user.

In this embodiment, the detection accuracy (reliability) of the facedirection is calculated on the basis of detection results of the wearingposition of the camera body 1 and the face direction. When thereliability is high, the face direction is largely reflected to theobservation direction. When the reliability is low, a factor other thanthe face direction is largely reflected to the observation direction.Thereby, the user's intention can be reflected to the image pickupoperation.

FIG. 38 is a flowchart showing an observation direction determinationprocess according to this embodiment in obtaining the face directionthat is executed instead of the process in the step S6004 in FIG. 35.This process is executed by the overall control CPU (anobservation-direction determination unit) 101.

In a step S7001, the overall control CPU (a first-observation-directioncalculation unit and a reliability calculation unit) calculates a facedirection reliability T_(n) on the basis of the face direction (a firstobservation direction) θ_(n) obtained by the face direction detectionunit 20 in picking up the image of the frame n.

The face direction reliability T_(n) is calculated as follows. First,the face direction θ_(n) is divided into three components, a facedirection θ_(yaw), a face direction θ_(pitch), and a face directionθ_(roll). The face direction θ_(yaw) is a rotation component of facemovement in the lateral direction. The face direction θ_(pitch) is arotation component of the face movement in the vertical direction. Theface direction θ_(roll) is a rotation component of a tilt of the head.

Since this embodiment assumes that the user wears the camera body on theuser's clavicular and detects the face direction from a position underthe face, the face direction reliability T_(n) (0≤T_(n)≤1) is found bythe following formula 701.

$\begin{matrix}{T_{n} = \frac{1}{1 + {{❘{\tan\left( {2 \cdot \theta_{yaw}} \right)}❘} \cdot {❘{{\tan\left( {2 \cdot \theta_{pitch}} \right)}{❘ \cdot ❘}{\tan\left( {2 \cdot \theta_{roll}} \right)}❘}}}}} & {{Formula}701}\end{matrix}$

FIG. 39 shows the relationship between the face direction θ_(yaw) theface direction reliability T_(n). A graph in FIG. 39 shows that the facedirection reliability T_(n) becomes lower as the angle of the facedirection θ_(yaw) from the front becomes larger.

The face direction reliability T_(n) is calculated using the formula 701in this embodiment. In the meantime, the face direction reliability maybe obtained by weight-averaging values calculated by weighting the pastface direction reliabilities according to the detection accuracy of theface direction by the face direction detection unit 20 and the framerate of the detection. Moreover, the accuracy of pattern matching, thewearing position, etc. may be weighted in calculating the face directionreliability T_(n).

Moreover, in this embodiment, the face direction reliability thatestimates the observation direction is calculated by the formula 701.However, the calculation method of the face direction reliability is notlimited to this. For example, a face direction reliability adjustedaccording to the wearing point of the camera body 1 that is estimated bythe calibration in the second embodiment may be employed. Moreover, whenit is determined a detection accuracy is low in the calibration, theface direction reliability may be changed according to the detectionaccuracy. Furthermore, when the face direction is detected using machinelearning, a precision ratio may be reflected to the face directionreliability.

In a step S7002, the overall control CPU 101 finds the angular speedω_(n) of movement of the face. Specifically, the angular speed ω_(n) isfound by the following formula 702 using the face direction θn obtainedby the face direction detection unit 20 in picking up the image of theframe n, its face direction obtainment time t_(n), the face directionθ_(n−1) of the previous frame stored in the primary memory 103, and itsface direction obtainment time t_(n−1).

$\begin{matrix}{\omega_{n} = \frac{\theta_{n} - \theta_{n - 1}}{t_{n} - t_{n - 1}}} & {{Formula}702}\end{matrix}$

Although the angular speed ω_(n) is calculated using the informationabout the current frame and the information about the previous frame inthis embodiment, the angular speed may be found using one or more piecesof past information depending on the frame rate etc.

In a step S7003, the overall control CPU (an observation-directionestimation unit) 101 estimates the current face direction from thetransition of the past face directions stored in the primary memory 103.In this embodiment, a period including four previous frames based on acurrent frame is defined as a predetermined period. When the continuouschange of face direction in a certain direction is determined at threeor more times within the predetermined period, it is determined that theobservation direction can be estimated from the past face directions andangular speeds. Moreover, in this estimation, an estimated angular speedω_(ave) that is a weighted average of the angular speeds obtained fromthe past four frames is calculated by the following formula 703, and anestimated face direction θ_(ave) (first observation direction) iscalculated by the following formula 704. The calculations of theformulae 703 and 704 respectively correspond to processes a1 and a2shown in FIG. 40.

It should be noted that the length of the predetermined period and theweights of the weighted average used in the step S7003 may be changedaccording to the frame rate and the detection accuracy of the facedirection detection unit 20.

$\begin{matrix}{\omega_{ave} = \frac{\sum_{t = 1}^{3}\omega_{n - t}}{3}} & {{Formula}703}\end{matrix}$ $\begin{matrix}{\theta_{ave} = {\theta_{n - 1} + {\left( {t_{n} - t_{n - 1}} \right) \cdot \omega_{ave}}}} & {{Formula}704}\end{matrix}$

In a step S7004, the overall control CPU 101 estimates the observationdirection using internal information other than the information from theface direction detection unit 20 from among the information stored inthe primary memory 103. Specifically, in this embodiment, it isdetermined whether the object is currently tracked on the basis of anobject detection history. When it is determined that the object iscurrently tracked, an estimated observation direction θ_(sub) (a secondobservation direction) based on the movement of the object iscalculated. In this embodiment, a period including four previous framesbased on a current frame is defined as a predetermined period. When anobject that can be determined as the identical object is detected atthree or more times within the predetermined period, it is determinedthat the object is currently tracked. The criterion of the objecttracking determination may be changed corresponding to the cycle andaccuracy of the detection by the overall control CPU 101. Since anobject tracking detection technique is publicly known, its detaileddescription is omitted.

Although the internal information used for the estimation of theobservation direction in the step S7004 is the object detection historyin this embodiment, it is not limited to this. For example, theobservation direction may be estimated according to the wearing positionand performance of the camera body 1 using face information of the usercaptured by the image pickup unit 40 or the information about movementand posture of the camera body 1 detected by the angular speed sensor107 and the acceleration sensor 108. Moreover, when there is an objectregistered beforehand, the overall control CPU (a third observationdirection estimation unit) 101 may determine the direction of the objectregistered beforehand in the newest pickup image as the estimatedobservation direction θ_(sub) as with the step S6006 in the sixthembodiment.

In a step S7005, the overall control CPU 101 storesface-direction-detection related information into the primary memory 103as a history. The face-direction-detection related information includesthe angular speed con of movement of the face generated in the stepS7002, the face direction reliability T_(n) calculated in the stepS7001, the face direction θ_(n) detected by the face direction detectionunit 20, the face direction obtainment time t_(n), and the generatedtime points of these values.

In a step S7006, the overall control CPU 101 determines whether the facedirection reliability T_(n) calculated in the step S7001 is equal to ormore than a predetermined value. When the face direction reliabilityT_(n) is equal to or more than the predetermined value, it is determinedthat the face direction reliability is high and the process proceeds toa step S7009. In the step S7009, the overall control CPU 101 determinesthe face direction as the current observation direction θ′_(n). Then,the process proceeds to the step S7013.

In the meantime, when the face direction reliability T_(n) calculated inthe step S7001 is less than the predetermined value (NO in the stepS7006), the process proceeds to a step S7007. In the step S7007, it isdetermined that whether the estimated face direction θ_(ave) can becalculated in the step S7003 and whether |θ_(n)−θ_(ave)| is equal to orless than a predetermined angle. When the two conditions are satisfied,the process proceeds to a step S7010. In this embodiment, thepredetermined angle is set to π/8 in the determination.

In the step S7010, the overall control CPU (a firstobservation-direction estimation unit) 101 determines the currentobservation direction θ′_(n) using the face direction θn, the estimatedobservation angle θ_(ave), and the face direction reliability T_(n). Inthis embodiment, the current observation direction θ′n is calculated bythe following formula 705, and the process proceeds to the step S7013.The calculation of the formula 705 corresponds to a process b1 shown inthe in FIG. 40. As shown in FIG. 39, the face direction reliabilityT_(n) becomes higher as the absolute value of the face angle θ_(yaw)becomes smaller. Accordingly, when the absolute value of the face angleθ_(y)aw is small, the face direction θ_(n) is largely reflected to thecurrent observation direction θ′_(n) as shown by the formula 705. In themeantime, when the absolute value of the face angle θ_(yaw) is large,the factor other than the face direction θn (specifically, the estimatedface direction θ′_(ave)) is largely reflected to the current observationdirection θ′n as shown by the formula 705.

θ′_(n) =T _(n)·θ_(n)+(1−T _(n))·θ_(ave)  Formula 705

When the above-mentioned conditions are not satisfied in the step S7007,the process proceeds to a step S7008. In the step S7008, it isdetermined whether the estimated observation direction θ_(sub) can becalculated and whether |θ_(n)−θ_(sub)| is equal to or smaller than apredetermined angle. When the conditions in the step S7008 aresatisfied, the process proceeds to a step S7011. In this embodiment, thepredetermined angle is set to π/8 in the determination as with the stepS7010.

In the step S7011, the overall control CPU (a secondobservation-direction estimation unit) 101 determines the currentobservation direction θ′_(n) using the face direction θ_(n), theestimated observation direction θ_(sub), and the face directionreliability T_(n). In this embodiment, the current observation directionθ′n is found by the following formula 706, and the process proceeds tothe step S7013. As shown in FIG. 39, the face direction reliabilityT_(n) becomes higher as the absolute value of the face angle θ_(yaw)becomes smaller in the same manner as the step S7010. Accordingly, whenthe absolute value of the face angle θ_(yaw) is small, the facedirection θ_(n) is largely reflected to the current observationdirection θ′_(n) as shown by the formula 706. In the meantime, when theabsolute value of the face angle θ_(yaw) is large, the factor other thanthe face direction θ_(n) (specifically, the estimated observationdirection θ_(sub)) is largely reflected to the current observationdirection θ′_(n)

θ′_(n) =T _(n)·θ_(n)+(1−T _(n))·θ_(sub)  Formula 706

When the above conditions are not satisfied in the step S7008, it isdetermined that reliable observation direction cannot be obtained in thepresent situation, and the process proceeds to a step S7012. In theS7012, the current observation direction θ′_(n) is determined by movingthe previous observation direction θ′_(n−1) with inertia based on thetransition of the past observation directions and the field angle iswidened than the prescribed field angle. Then, the process proceeds tothe step S7013. This reduces a possibility that the user misses pickingup an object that the user intends.

Although the calculation method of the current observation directionθ′_(n) is switched according to the face direction reliability T_(n) andthe detection state of the object in this embodiment, it is not limitedto this. For example, when the estimated face direction θ_(ave) andestimated observation direction θ_(sub) are calculated, theirreliabilities (estimated direction reliabilities) may be alsocalculated. In such a case, the calculated observation direction θ′_(n)can be corrected according to the calculated reliabilities.

Moreover, since the possibility that the user misses picking up anobject that the user intends becomes high when the calculatedreliabilities are not more than the predetermined value, it ispreferable to widen the field angle than the prescribed field angle.Moreover, in such a case, the process may proceed to the step S7012.After that, when one of the calculated reliabilities will become largerthan the predetermined value, it is preferable to restore the widenedfield angle to the prescribed field angle.

As a result of the process in FIG. 38, when the face directionreliability T_(n) is high, the face direction θ_(n) is determined as thecurrent observation direction θ′_(n). In the meantime, when the facedirection reliability T_(n) is low, the current observation directionθ′_(n) (recording direction) is determined using the face directionsobtained under the high face direction reliability T_(n), the factorother than the face direction, or the like according to the situation.Furthermore, the field angle is widened if needed.

Namely, when the low detection accuracy of the face direction isestimated because of the low face direction reliability T_(n) theestimated face direction θ_(ave) or the estimated observation directionθ_(sub) is used. Thereby, a situation where an image that the user doesnot intend is picked up because of failure of face direction detectioncan be prevented.

Next, an eighth embodiment will be described. In the eighth embodiment,a method to wear the camera body 1 at a stable position will bedescribed using FIG. 41A through FIG. 45G. This embodiment is describedas a derivation from the first embodiment basically. Accordingly,configurations of the camera system in the eighth embodiment that areidentical to the configurations of the camera system in the firstembodiment are indicated by the same reference numerals and duplicateddescriptions are omitted. A different configuration will be described byadding details.

The angle adjustment of the connection members 80L and 80R (neck hangingmembers) is described first. FIG. 41A, FIG. 41B, and FIG. 41C areenlarged side views showing the image-pickup/detection unit 10. Althoughthe following description exemplifies the left connection member 80L,the right connection member 80R is adjusted similarly.

FIG. 41A is a view showing a state where the connection member 80L isset in a standard position Ax0. FIG. 41B is a view showing a state wherethe connection member 80L has rotated by an angle θA1 with respect tothe standard position Ax0 around a rotation axis OA. FIG. 41C is aschematic view showing a mechanical structure inside the angle holdingmember 81L that can be seen when an exterior of the angle holding member81L is removed.

As shown in FIG. 41C, an angle adjustment mechanism (neck-hanging-angleadjustment mechanism) 8100 is arranged inside the angle holding member81L.

The angle adjustment mechanism 8100 consists of an angle adjustment cam8101 that adjusts the angle of the angle holding member 81L with respectto the image-pickup/detection unit 10 and a latching member 8102 thatlatches the angle adjustment cam 8101. It should be noted that therotation axis OA of the angle holding member 81L agrees with the centerof the angle adjustment cam 8101.

The latching member 8102 is energized to the angle adjustment cam 8101with a spring (not shown). While the angle adjustment button 85L (FIG.2F) is pressed, the energization is released and the latching member8102 can separate from the angle adjustment cam 8101. Namely, only whilethe angle adjustment button 85L is pressed, the angle holding member 81Lof the connection member 80L becomes rotatable with respect to theimage-pickup/detection unit 10.

When rotating the angle holding member 81L with respect to theimage-pickup/detection unit 10 while pressing the angle adjustmentbutton 85L, the user is able to adjust the connection member 80L fromthe standard position Ax0 (FIG. 41A) to a position Ax1 (FIG. 41B).

Although this embodiment employs a stepped adjustment mechanism, whichconsists of the angle adjustment cam 8101 and latching member 8102, asthe mechanism that holds the angle of the angle holding member 81L withrespect to the image-pickup/detection unit 10, a stepless adjustmentmechanism using sliding resistance may be employed.

Moreover, although this embodiment employs the configuration that theuser rotates the angle holding member 81L while pressing the angleadjustment button 85L, it is not limited to this. For example, aconfiguration that does not need the angle adjustment button 85L may beused. Such a configuration allows rotation of the angle holding member81L when external force more than a threshold is applied. For example, aball may be used instead of the latching member 8102 and slidingresistance may be used.

FIG. 42A, FIG. 42B, and FIG. 42C are side views showing a state wherethe user wears the camera body 1. FIG. 42A is a view showing the userwho wears the camera body 1 of which the connection member 80L is set inthe standard position Ax0 and the band part 82L is elongated. FIG. 42Bis a view showing the user who wears the camera body 1 of which theconnection member 80L is set in the standard position Ax0 and the bandpart 82L is shortened. FIG. 42C is a view showing the user who wears thecamera body 1 of which the connection member 80L is set in the positionAx1 and the band part 82L is shortened.

As shown in FIG. 42A and FIG. 42C, when the relationship between theposition of the connection member 80L and the length of the band part82L is suitable for the user, the image pickup lens 16 is directed tothe user's front. In the meantime, as shown in FIG. 42B, when therelationship between the position of the connection member 80L and thelength of the band part 82L is not suitable for the user, the imagepickup lens 16 is not directed to the user's front. In the case in FIG.42B, the optical axis of the image pickup lens 16B is directed upward.

In this way, since the connection member 80L is constituted to beadjustable in its position, the user is able to wear the camera body 1so that the optical axis of the image pickup lens 16 will beapproximately parallel to a visual line in a user's natural state. Whenthe optical axis of the image pickup lens 16 matches the horizontaldirection when the user wears the camera body 1 at the suitableposition, suitable wearing is available similarly.

Next, the angle adjustment of the chest contact pads 18 a and 18 b willbe described. FIG. 43A, FIG. 43B, and FIG. 43C are enlarged side viewsshowing the image-pickup/detection unit 10 without showing theconnection members 80L and 80R. Although the following descriptionexemplifies the left chest contact pad 18 a, the right chest contact pad18 b is adjusted similarly.

FIG. 43A is a view showing a state where the chest contact pad 18 a isset in a standard position Bx0. FIG. 43B is a view showing a state wherethe chest contact pad 18 a has rotated by an angle θB1 with respect tothe standard position Bx0 around a rotation axis OB. FIG. 43C is aschematic view showing a mechanical structure inside theimage-pickup/detection unit 10 that can be seen when an exterior of theimage-pickup/detection unit 10 is removed.

As shown in FIG. 43C, a contact-angle adjustment mechanism 8200 isarranged inside the image-pickup/detection unit 10. The contact-angleadjustment mechanism 8200 consists of an angle adjustment cam 8201 thatadjusts the angle of the chest contact pad 18 a with respect to theimage-pickup/detection unit 10 and a latching member 8202 that latchesthe angle adjustment cam 8201. The rotation axis OB shown in FIG. 43Athrough FIG. 43C is a rotation center of the chest contact pad 18 a.

The latching member 8202 is energized to the angle adjustment cam 8201with a spring (not shown). While an angle adjustment button 8203 ispressed, the energization is released and the latching member 8202 canseparate from the angle adjustment cam 8101. Namely, only while theangle adjustment button 8203 is pressed, the chest contact pad 18 abecomes rotatable with respect to the image-pickup/detection unit 10.

When rotating the chest contact pad 18 a with respect to theimage-pickup/detection unit 10 while pressing the angle adjustmentbutton 8203, the user is able to adjust the chest contact pad 18 a fromthe standard position Bx0 to a position Bx1.

Although this embodiment employs a stepped adjustment mechanism, whichconsists of the angle adjustment cam 8201 and latching member 8202, asthe mechanism that holds the angle of the chest contact pad 18 a withrespect to the image-pickup/detection unit 10, a stepless adjustmentmechanism using sliding resistance may be employed.

Moreover, although this embodiment employs the configuration that theuser rotates the chest contact pad 18 a while pressing the angleadjustment button 8203, it is not limited to this. For example, aconfiguration that does not need the angle adjustment button 8203 may beused. Such a configuration allows rotation of the chest contact pad 18 awhen external force more than a threshold is applied. For example, aball may be used instead of the latching member 8202 and slidingresistance may be used.

FIG. 44A, FIG. 44B, and FIG. 44C are side views showing states whereusers wear the camera body 1 without showing the connection members 80Land 80R. FIG. 44A shows a state where a user whose chest is steep wearsthe camera body 1 of which the chest contact pad 18 a is set at thestandard position Bx0. FIG. 44B shows a state where a user whose chestis gentle wears the camera body 1 of which the chest contact pad 18 a isset at the standard position Bx0. FIG. 44C shows a state where a userwhose chest is gentle wears the camera body 1 of which the chest contactpad 18 a is set at the position Bx1.

As shown in FIG. 44A and FIG. 44C, when the position of the chestcontact pad 18 a is suitable for the inclination of the chest of theuser, the chest contact pad 18 a contacts the chest of the user in awide area. In the meantime, as shown in FIG. 44B, when the position ofthe chest contact pad 18 a is not suitable for the inclination of thechest of the user, the chest contact pad 18 a contacts the chest of theuser in only a few areas. When the area in which the chest contact pad18 a contacts the chest of the user becomes small as shown in FIG. 44B,the image-pickup/detection unit 10 will deviate from the user's bodyeasily due to movement of the user's body, which causes great blur in apickup image.

Since the chest contact pad 18 a is constituted so as to adjust itsangle easily, the user is able to wear the camera body 1 so that thechest contact pad 18 a contacts the user's chest in a wide area, whichreduces blur in a pickup image.

Although the chest contact pad 18 a is arranged in theimage-pickup/detection unit 10 in this embodiment, it may be arranged inthe connection member 80L. Even in such a case, similar effect isobtained. In this case, for example, a mechanism similar to the angleadjustment mechanism 8100 shown in FIG. 41C will be arranged inside theconnection member 80L as a mechanism that adjusts the angle of the chestcontact pad 18 a with respect to the connection member 80L.

Next, the configurations of the band part 82L and electric cable 84 willbe described in detail. As described in the first embodiment, thebattery unit (power source unit) 90 and image-pickup/detection unit 10of the camera body 1 are the separate modules that are electricallyconnected through the electric cable 84.

If the electric cable 84 and the band part 82L are separated, theappearance of the camera body 1 deteriorates and a wearing operation ofthe camera body 1 becomes troublesome. It is not preferable.Accordingly, it is preferable to integrate the band part 82L and theelectric cable 84. In the meantime, an integrated configuration is notlimited to the configuration shown in FIG. 2B.

FIG. 45A through FIG. 45G are views showing various configurations ofthe band part 82L and the connection surface 83L that is a section ofthe electric cable 84 united with the band part 82L. FIG. 45A throughFIG. 45C show configurations where the electric cable 84 is constitutedby a flexible substrate (FPC). FIG. 45D through FIG. 45G showconfigurations where the electric cable 84 is constituted by the thinwire cable.

FIG. 45A and FIG. 45D show configurations where the electric cable 84 isembedded inside the band part 82L viewed from the connecting face 83L.In this case, the band part 82L is preferably made from elasticmaterials, such as silicone rubber, elastomer, rubber, and a plastic,that enable injection molding. For example, the electric cable 84 isinserted into the electric cable 84 at the time of the injectionmolding. Otherwise, the band part 82L may be constituted from twocomponents. In such a case, the electric cable 84 is sandwiched betweenthe components of the band part 82L and they are united by adhesive orheat welding. Manufacturing methods are not limited to the above twomethods. Any other methods can be employed as long as the band part 821and the electric cable 84 are united as shown in FIG. 45A and FIG. 45D.

FIG. 45B, FIG. 45C, and FIG. 45E show configurations where the electriccable 84 is connected to the outer side of the band part 82L viewed fromthe connecting face 83L. FIG. 45B shows a configuration where theelectric cable 84 is adhered to the band part 82L. The band part 82L hasno specific configuration to unite with the electric cable 84. Thisconfiguration can be manufactured at low cost. When the electric cable(FPC in this case) 84 appears externally, the appearance of product canbe improved by painting the FPC or by covering the FPC with a film.Moreover, when the electric cable (FPC in this case) 84 is arranged onthe inner side (neck side) of the band part 82L in the configurationshown in FIG. 45B, wearing feeling can be improved by painting the FPCor by covering the FPC with a film.

FIG. 45C and FIG. 45E show configurations where a concave portion 83 ais formed in the band part 82L viewed from the connecting face 83L inorder to unite with the electric cable 84. The electric cable 84 isarranged in the concave portion 83 a. In this case, when the concaveportion 83 a is arranged on the inner side (neck side) of the band part82L, good appearance can be maintained and good wearing feeling is alsokept without performing a special process because the electric cable 84is stored in the concave part 83 a and does not directly contact theuser's neck. Furthermore, since the concave shape 83 a does not needadditional cost if it is designed before manufacturing, it has a meritin respect of cost.

FIG. 45F and FIG. 45G show configurations where the electric cable 84 isembedded inside the band part 82L viewed from the connecting face 83L.FIG. 45F shows a configuration where the electric cable 84 consists ofsingle line. FIG. 45G shows a configuration where the electric cable 84consists of three lines. A characteristic feature of the configurationsin FIG. 45F and FIG. 45G is that cross-sectional area in the connectingface 83L of the band part 82L is secured. That is a different point fromthe configurations in FIG. 45A and FIG. 45D. The cross-sectional area inthe connecting face 83L of the band part 82L gives influence to twistrigidity and bending rigidity. And such rigidity influences stabilityfor the image-pickup/detection unit 10 to be continuously stabilized ata fixed position of the user's bod when the user wears the camera body1. That is, the stability of the image-pickup/detection unit 10 improvesas the cross-sectional area in the connecting face 83L increases becausethe twist rigidity and flexural rigidity become strong.

The projection side of the electric cable 84 is preferably arranged atthe outer side of the band part 82L in order to obtain comfortablewearing feeling. The configurations in FIG. 45F and FIG. 45G expose theprojection side to the outer appearance but ensure the rigidity of theband part 82L.

As mentioned above, the configuration in FIG. 45C or FIG. 45E has anadvantage if a priority is given to balance between the appearance andthe wearing feeling. If a priority is given to the cost or rigidity,another configuration in FIG. 45A, FIG. 45B, FIG. 45D, FIG. 45F, or FIG.45G can be employed.

Next, a ninth embodiment will be described. In the ninth embodiment, amodified example of the camera system including the camera body 1 willbe described using FIG. 46A and FIG. 46B. This embodiment is describedas a derivation from the first embodiment basically. Accordingly,configurations of the camera system in the ninth embodiment that areidentical to the configurations of the camera system in the firstembodiment are indicated by the same reference numerals and duplicateddescriptions are omitted. A different configuration will be described byadding details.

The display apparatus 800 in the first embodiment uses the general smartphone. There are various smart phones in the commercial scene and theirarithmetic capacities are also various. For example, the displayapparatus 800 in the first embodiment has relatively high arithmeticcapacity. Accordingly, when the camera body 1 transfers the image of therecording direction that is extracted from the superwide-angle image tothe display apparatus 800, the information required for the opticalcorrection process or the image stabilization process is added to theimage. The display apparatus 800 in the first embodiment performs thedistortion correction process and the image stabilization process basedon the added information. However, such processes are hard for a smartphone having relatively low arithmetic capacity.

The camera system of this embodiment is provided with a camera body 1′including an image pickup apparatus and a display apparatus 9800 ofwhich arithmetic capacity is lower than that of the display apparatus800. When the camera body 1′ has performed the processes from thepreparation process to the primary recording process (the steps S100through S600 in FIG. 7A), the camera body 1′ performs the opticalcorrection process and the image stabilization process (the steps S800and S900) without performing the transmission process to the displayapparatus (the step S700). After that, the camera body 1′ performs aprocess that transfers the image that has been subjected to theprocesses in the steps S800 and S900 to the display apparatus 9800.

In the meantime, the display apparatus 9800 performs the secondaryrecording process (S1000) to the image from the camera body 1′ withoutperforming the processes in the steps S800 and S900.

Hereinafter, the camera system of this embodiment will be describedspecifically. FIG. 46A is a block diagram showing a hardwareconfiguration of the display apparatus 9800 connected to the camera body1′ including the image pickup apparatus according to this embodiment.

In FIG. 46A, hardware configurations of the display apparatus 9800 thatare identical to the hardware configurations of the display apparatus800 according to the first embodiment shown in FIG. 6 are indicated bythe same reference numerals and duplicated descriptions are omitted.

The display apparatus 9800 has a display-apparatus controller 9801instead of the display-apparatus controller 801 of the display apparatus800, and does not have the face sensor 806.

The display-apparatus controller 9801 is constituted by a CPU of whicharithmetic capacity is lower than the CPU that constitutes thedisplay-apparatus controller 801 (FIG. 6). Moreover, capacities of theinternal nonvolatile memory 812 and the primary memory 813 may be lowerthan that in the first embodiment.

FIG. 46B is a functional block diagram showing the camera body 1′. InFIG. 46B, function blocks of the camera body 1′ that are identical tothe hardware blocks of the camera body 1 according to the firstembodiment shown in FIG. 4 are indicated by the same reference numeralsand duplicated descriptions are omitted.

The functional block diagram in FIG. 46B differs from the function blockdiagram in FIG. 4 in the following points. That is, anoptical-correction/image-stabilization unit 9080 that performs theoptical correction process and image stabilization process is provided.And an overall control CPU 9101 is provided instead of the overallcontrol CPU 101. Moreover, the transmission units 70 communicates withthe display apparatus 9800 instead of the display apparatus 800.

That is, in this embodiment, the optical-correction/image-stabilizationunit 9080 of the overall control CPU 9101 performs the opticaldistortion correction and the image stabilization process using theoptical correction values and gyro data. Accordingly, since thetransmission unit 70 transmits a video file to the display apparatus9800 after applying the optical correction process and imagestabilization process in this embodiment, a data amount of the videofile in this embodiment is smaller than that of the video file 1000 thatthe transmission unit 70 transfers to the display apparatus 800 in thefirst embodiment.

Moreover, the display apparatus 9800 does not need the high arithmeticcapacity equal to that of the display apparatus 800 because it does notperform the processes in the steps S800 and S900. Moreover, the imagepicked up by camera body 1′ can be seen by the simplified display device(an appreciation device) 900 like a smart watch.

Next, a tenth embodiment will be described. In the tenth embodiment, amodified example of the camera system including the camera body 1 willbe described using FIG. 47 and FIG. 48. This embodiment is described asa derivation from the first embodiment basically. Accordingly,configurations of the camera system in the tenth embodiment that areidentical to the configurations of the camera system in the firstembodiment are indicated by the same reference numerals and duplicateddescriptions are omitted. A different configuration will be described byadding details.

In the ninth embodiment, the camera body 1′ needs high performanceinstead of using the display apparatus 9800 having the low arithmeticcapacity. When the performance of the camera body is improved, the costof the overall controller CPU and its peripheral devices may rise andheat generation due to a processing load may occur. Accordingly, in thetenth embodiment, a configuration that decreases the arithmetic capacityof the camera body and increases the arithmetic capacity of the displayapparatus will be described.

FIG. 47 is a functional block diagram showing a camera system of thisembodiment including a camera body 1001 and a display apparatus 1080. InFIG. 47, function blocks of the camera body 1001 that are identical tothe hardware blocks of the camera body 1 according to the firstembodiment shown in FIG. 4 or the camera body 1′ according to the ninthembodiment shown in FIG. 46B are indicated by the same referencenumerals and duplicated descriptions are omitted.

The functional block diagram shown in FIG. 47 differs from FIG. 4 andFIG. 46B greatly in that the display apparatus 1080 is provided with arecording-direction/field-angle determination unit 1083, an imageextraction/development unit 1084 that extracts and develops an image,and an optical-correction/image-stabilization unit 1085 that performsthe optical correction process and image stabilization process.

A face-image primary processing unit 1030 that processes the face imagedetected by the face direction detection unit 20, a main-image primaryprocessing unit 1050 that processes the main image picked up by theimage pickup unit 40, and an image combination unit 1055 that combinesthese images are added to the camera body 1001. Therecording-direction/field-angle determination unit 1083 and imageextraction/development unit 1084 are moved to the display apparatus1080. And an image separation unit 1082 is added to the displayapparatus 800. Moreover, a reception unit 1081 of the display apparatus1080, which is not shown in FIG. 4 and FIG. 46B, is added to FIG. 47.

An order of the process will be described using a flowchart in FIG. 48.A process in the flowchart in FIG. 48 that is equivalent to a process inthe flowchart in FIG. 7A is indicated by a step number that is obtainedby adding 10000 to the original step number (i.e., “10” is added toupper two digits), and a duplicated description is omitted. Moreover, inorder to assist the description, a reference numeral of an apparatus inFIG. 47 that executes a process in each step is shown on a right side ofeach step in FIG. 48. That is, steps S10100 through S10700 in FIG. 48are executed by the camera body 1001, and steps S10710 through S10950are executed by the display apparatus 10800.

In FIG. 7A in the first embodiment, the face direction detection processis performed in the step S200 after the preparation process in the stepS100. In this embodiment, a face image pickup process in a step S10200and a main image pickup process in a step S10400 are executed inparallel after a preparation process in a step S10100. Next, two imagedata picked up in steps S10200 and S10400 are combined in an imagecombination process in a by S10450. Several kinds of combination methodsare considered. The two images may be combined into one video file ortwo images may be mutually associated so as not to deviate frames ofdata of the two images.

This embodiment is described on the basis of the method that combinestwo images into one video file. In the step S10450, a combined imagethat is primarily recorded is wirelessly transmitted to the displayapparatus 10180 in a step S10700.

The steps from the step S10710 is executed by the display apparatus1080. In the step S10710, the image combined in S10450 is againseparated into a face pickup image and a main pickup image.Subsequently, in a step S10720, the face direction detection processthat estimates an observation direction from the separated face pickupimage is executed. It should be noted that the contents of the facedirection detection process have been described in the first embodimentusing FIG. 7C.

In a step S10730, a recording-direction/area determination process isexecuted. In a step S10750, a recording-area development process isexecuted. Specifically, an image is extracted from the main pickup imageseparated in the step S10710 on the basis of therecording-direction/field-angle information determined in the stepS10730 and the extracted area is developed. In a step S10800, theoptical correction process that corrects optical aberrations is appliedto the image that is extracted and developed in the step S10750. Theimage stabilization process is performed in a step S10900.

Also in this embodiment, the order of the steps S10800 and S10900 may beinverted. That is, the image stabilization process may be executed inadvance of the optical correction process.

In the step S10950, the display-apparatus controller (a video recordingunit) executes a secondary recording process to record the image intothe large-capacity nonvolatile memory 814 after applying the opticalcorrection process in the step S10800 and the image stabilizationprocess in the step S10900. And then, this process is finished.

In this embodiment, since the combination data that combines the mainpickup image and the face pickup image is transmitted in the stepS10700, the process with the camera body 1001 is simplified, whichenables reduction of the cost and reduction of heat generation. Itshould be noted that the gyro data and posture data that are output fromthe angular speed sensor 107 and the acceleration sensor 108 may betransmitted to the display apparatus 1080 in the step S10700 as with thefirst embodiment.

Next, an eleventh embodiment will be described. In the eleventhembodiment, a modified example of the camera system including the camerabody 1 will be described using FIG. 49 and FIG. 50. This embodiment isdescribed as a derivation from the first embodiment basically. Since thebasic configuration of this embodiment is similar to that of the tenthembodiment, configurations of the camera system in the eleventhembodiment that are identical to the configurations of the camera systemin the tenth embodiment are indicated by the same reference numerals andduplicated descriptions are omitted. A different configuration will bedescribed by adding details.

In the tenth embodiment, the configuration that decreases the arithmeticcapacity of the camera body and increases the arithmetic capacity of thedisplay apparatus 1080 is described. Although this configuration canreduce a load of the overall control CPU of the camera body, the amountof data transmitted from the transmission unit 70 increases, which mayleave issues like heat generation.

Moreover, some controllers loaded on recently developed cameras includea circuit specialized in image processing. For example, a controllerincluding a circuit for a face direction detecting function needed forthis disclosure can be developed. Use of such a controller prevents thecost from increasing and reduces power consumption. In this embodiment,such a controller is employed. The camera body 1101 performs until theface direction detection process and the recording-direction/areadetermination process, adds the result data of these processes to themain pickup image, and transmits the main pickup image to the displayapparatus 1180. And the display apparatus 1180 performs therecording-area development process to extract and develop an image.

FIG. 49 is a functional block diagram showing a camera system of thisembodiment including the camera body 1101 and the display apparatus1180. In FIG. 49, function blocks of the camera body 1101 that areidentical to the hardware blocks of the camera body 1 according to thefirst embodiment, the camera body 1′ according to the ninth embodiment,or the camera body 1001 according to the tenth embodiment are indicatedby the same reference numerals and duplicated descriptions are omitted.

The camera system shown in FIG. 49 is different from FIG. 4 and FIG. 46Bin that the display apparatus 1180 is equipped with an imageextraction/development unit 1184 that extracts an image and develops itand an optical-correction/image-stabilization unit 1185 that performs anoptical correction process and an image stabilization process. Moreover,since the image extraction/development unit is moved to the displayapparatus 1180, the overall control CPU 101 is equipped with until therecording-direction/field-angle determination unit 30 but is notequipped with the image extraction/development unit 50. An informationcombination unit 1150 that combines the recording-direction/field-angleinformation to the main pickup image output from the image pickup unit40 is added to the camera body 1101. The image extraction/developmentunit 1184 is moved to the display apparatus 1180 as with the tenthembodiment. An information separation unit 1182 is added to the displayapparatus 1180. Moreover, a reception unit 1181, which is not shown inFIG. 4 and FIG. 46B, is added to FIG. 49 in the same manner as the tenthembodiment.

An order of the process will be described using a flowchart in FIG. 50.A process in the flowchart in FIG. 50 that is equivalent to a process inthe flowchart in FIG. 7A is indicated by a step number that is obtainedby adding 11000 to the original step number (i.e., “11” is added toupper two digits), and a duplicated description is omitted. Moreover, inorder to assist the description, a reference numeral of an apparatus inFIG. 49 that executes a process in each step is shown on a right side ofeach step in FIG. 50. That is, steps S11100 through S11700 in FIG. 50are executed by the camera body 1101, and steps S11710 through S11950are executed by the display apparatus 11800.

In the tenth embodiment, the face image pickup process in the stepS10200 and the main image pickup process in the step S10400 are executedin parallel, and the two image data ae combined in the step S10450. Inthe meantime, in the eleventh embodiment, after the face image is pickedup in the step S11200, the recording-direction/area determinationprocess is executed in a step S11400 and outputsrecording-direction/area data.

After that, the main image data picked up by the main image pickupprocess in a step S11300 that is executed in parallel and therecording-direction/area data output in the step S11400 are combined ina step S11450.

Although several kinds of combination methods of therecording-direction/area data are considered. In this eleventhembodiment, the recording-direction/area data is recorded as metadatafor every fame of the main pickup image data. The configuration of themetadata is the same as the metadata shown in FIG. 15.

The main pickup image data generated in the step S11450 is primarilyrecorded in a step S11600 and is wirelessly transmitted in the stepS11700 to the display apparatus 11180.

The steps from the step S11710 is executed by the display apparatus1180. In the step S11710, the image data with the metadata generated inthe step S11450 is again separated into the main pickup image and therecording-direction/area data.

In the next step S11750, a recording-area development process isexecuted. Specifically, an image is extracted from the main pickup imageseparated in the step S11710 on the basis of therecording-direction/field-angle information and the extracted area isdeveloped.

In a step S11800, the optical correction process that corrects opticalaberrations is applied to the image that is extracted and developed inthe step S11750. The image stabilization process is performed in a stepS11900.

Also in this embodiment, the order of the steps S11800 and S11900 may beinverted. That is, the image stabilization process may be executed inadvance of the optical correction process.

In the step S11950, the display-apparatus controller (video recordingunit) executes a secondary recording process to record the image intothe large-capacity nonvolatile memory 814 after applying the opticalcorrection process in the step S11800 and the image stabilizationprocess in the step S11900. And then, this process is finished.

In this embodiment, since the main pickup image and therecording-direction/area data are combined as timed metadata in the stepS11450, the capacity of the image data transferred in the step S11700can be reduced, which can reduce the power consumption, the heatgeneration, and the load on the display apparatus 1180. It should benoted that the gyro data and posture data that are output from theangular speed sensor 107 and the acceleration sensor 108 may betransmitted to the display apparatus 1180 in the step S11700 as with thefirst embodiment.

Next, a twelfth embodiment will be described. In the twelfth embodiment,an image pickup direction is changed by mechanically driving thedirection of the image pickup unit will be described using FIG. 51Athrough FIG. 56. FIG. 51A is an external view showing a camera body 1220according to this embodiment.

A part in the twelfth embodiment that has the same function of a partthat has been already described in the first embodiment is indicated bythe same reference numeral and its description in this specification isomitted. The camera body 1220 is provided with an image-pickup/detectionunit 1221, the connection members 80L and 80R, and the battery unit 90.

FIG. 51B is a perspective view showing details of theimage-pickup/detection unit 1221 that is a part of the camera body 1220.The image-pickup/detection unit 1221 is provided with a main body 1210,a yaw drive shaft 1201, a yaw drive base 1202, a pitch drive shaft 1203,and an image pickup unit 40. The main body 1210 is provided with a powerswitch 11, an image pickup mode switch 12, a face direction detectionwindow 13, a start switch 14, a stop switch 15, and a yaw drive motor1204.

The yaw drive motor 1204 drives the yaw drive base 1202 in a yawdirection (lateral direction) through the yaw drive shaft 1201. The yawdrive base 1202 is provided with a pitch drive motor 1205. The pitchdrive motor 1205 drives the image pickup unit 40 in a pitch direction(vertical direction) through the pitch drive shaft 1203.

The image pickup unit 40 is provided with an image pickup lens 16 and asolid state image sensor (not shown). The image pickup lens 16 guideslight from an object and forms an image of the object on the solid stateimage sensor 42.

FIG. 51C is a perspective view showing a state where the image pickupunit 40 turns leftward by 30°. FIG. 51D is a perspective view showing astate where the image pickup unit 50 is directed downward by 30°. Asshown in FIG. 51C, when the yaw drive motor 1204 is driven, the partsfrom the yaw drive shaft 1201 rotates in the lateral direction, whichchanges the direction of the image pickup unit 40 in the lateraldirection. As shown in FIG. 51D, when the pitch drive motor 1205 isdriven, the parts from the pitch drive shaft 1201 rotates in thevertical direction, which changes the direction of the image pickup unit40 in the pitch direction.

FIG. 52 is a functional block diagram showing the camera body 1220according the twelfth embodiment. Hereinafter, the process executed bythe camera body 1220 will be described roughly using FIG. 52. In thefollowing description, only points changed from FIG. 4 will be describedand function blocks that are identical to the hardware blocks in FIG. 4are indicated by the same reference numerals and duplicated descriptionsare omitted.

As shown in FIG. 52, the camera body 1220 is provided with the facedirection detection unit 20, an image-pickup-unit drive unit 1230, theimage pickup unit 40, a development unit 1250, the primary recordingunit 60, the transmission unit 70, and the second controller 111. Thesefunctional blocks are achieved by control of the overall control CPU 101(FIG. 53) that controls the entire camera body 1220.

The face direction detection unit 20 detects a face direction, estimatesan observation direction, and passes it to the image-pickup-unit driveunit 1230. The image-pickup-unit drive unit 1230 can change the imagepickup direction and the field angle by performing various calculationon the basis of the observation direction estimated by the facedirection detection unit 20 and the outputs of the angular speed sensor107 and acceleration sensor 108.

The image pickup unit 40 converts the light from an object light into animage, forms a wide-angle image of the object and passes the image tothe age extraction/development unit 50. The development unit 1250develops the image from the image pickup unit 40, and passes the imageof the direction that the users is looking at to the primary recordingunit 60. The primary recording unit 60 passes the image to thetransmission unit 70 at a required timing. The transmission unit 70 iswirelessly connected with predetermined communication parties, such asthe display apparatus 800, a calibrator 850, and a simplified displaydevice 900, and communicates with these.

FIG. 53 is a block diagram showing a hardware configuration of thecamera body 1220 according to the twelfth embodiment. Hereinafter, onlydifferences from FIG. 5 of the first embodiment will be described. Asshown in FIG. 53, the camera body 1220 is provided with a phasedetection sensor 1206 and a motor drive circuit 1207. The phasedetection sensor 1206 detects phases of the pitch and yaw of the imagepickup unit 40 and outputs them to the overall control CPU 101.

The motor drive circuit 1207 is controlled by the overall control CPU101 and drives the image pickup unit 40 to a desired direction at adesired driving speed.

Hereinafter, how to use the camera body 1 and display apparatus 800 willbe described. FIG. 54 is a flowchart schematically showing an imagepickup/recording process according to this embodiment executed by thecamera body 1220 and display apparatus 800. In order to assist thedescription, a reference numeral of an apparatus in FIG. 52 thatexecutes a process in each step is shown on a right side of each step inFIG. 54.

When the power switch 11 is set to ON and power of the camera body 1turns ON, the overall control CPU 101 is activated and reads a bootprogram from the internal nonvolatile memory 102. After that, in a stepS100, the overall control CPU 101 executes a preparation process thatperforms setting of the camera body 1 before an image pickup operation.

In a step S200, the face direction detection process that estimates anobservation direction based on a face direction detected by the facedirection detection unit 20 is executed. This process is executed at apredetermined frame rate.

In a step S12300, the image-pickup-unit drive unit 1230 performs animage-pickup-unit drive process to calculate a dive amount of the imagepickup unit 40 and to drivingly control the image pickup unit 40.Details of the image-pickup-unit drive process will be mentioned laterusing FIG. 55.

In a step S400, the image pickup unit 40 picks up an image and generatespickup image data. In a step S12500, the development unit 1250 executesthe recording-area development process that applies the developmentprocess to the image data generated in the step S400. Details of thedevelopment process will be mentioned later using FIG. 56.

In a step S600, the primary recording unit (image recording unit) 60executes the primary recording process that stores the image developedin the step S12500 into the primary memory 103 as image data. In thestep S700, the transmission unit 70 executes a transmission process tothe display apparatus 800 that wirelessly transmits the image primarilyrecorded in the step S600 to the display apparatus 800 at a designatedtiming.

The steps from the step S800 is executed by the display apparatus 800.In the step S800, the display-apparatus controller 801 executes anoptical correction process that corrects optical aberrations of theimage transferred from the camera body 1 in the step S700.

In a step S900, the display-apparatus controller 801 applies the imagestabilization process to the image of which optical aberrations havebeen corrected in the step S800. It should be noted that the order ofthe step S800 and the step S900 may be inverted. That is, the imagestabilization process may be executed in advance of the opticalcorrection process.

In a step S1000, the display-apparatus controller (video recording unit)801 executes a secondary recording process that records the image towhich the optical correction process in the step S800 and the imagestabilization process in the step S900 have been applied into thelarge-capacity nonvolatile memory 814. And then, the display-apparatuscontroller 801 finishes this process.

FIG. 55 is a flowchart showing the subroutine the image-pickup-unitdrive process in the step S1230 in FIG. 54. In a step S12301, theoverall control CPU 101 obtains outputs of the angular speed sensor 107,the acceleration sensor 108, and the phase detection sensor 1206.

In a step S12302, the overall control CPU 101 calculates the controlamounts of the pitch drive motor 1205 and yaw drive motor 1204 on thebasis of the observation direction (direction vector) recorded in thestep S212 (FIG. 7C), and the outputs of the various sensors obtained inthe step S12301. At this time, the overall control CPU 101 performsfeedback control and image stabilization control aimed to a targetvalue. The control amounts can be calculated by a known control process.

The overall control CPU 101 controls the motor drive circuit 1207 (astep S12303) on the basis of the control amounts calculated in the stepS12302 to drive the pitch drive motor 1205 and yaw drive motor 1204 (astep S12304), and finishes this process (a step S12305).

FIG. 56 is a flowchart showing a subroutine of the recording-areadevelopment process in the step S12500 in FIG. 54. Difference from FIG.7E is that the flowchart in FIG. 56 does not have the step S502 (i.e.,does not obtain Xi, Yi, WXi, and WYi) and proceeds to the step S503immediately after obtaining the entire area Raw data in the step S501.

When the video image mode is selected by the image pickup mode switch12, the processes in the steps S200 and S12300 and the processes in thestep S400, S12500, and S600 are executed in parallel. The drive of theimage pickup unit 40 is continued on the basis of the detection resultof the observation direction while continuing the image pickup operationby the image pickup unit 40.

When the image pickup operation is executed under the above-mentionedconfiguration and control, the user is able to pick up an image whiledirecting the image pickup unit 40 toward the user's observationdirection without being conscious of the image pickup operation.

Next, a thirteenth embodiment will be described. The thirteenthembodiment detects a face direction using machine learning, such as DeepLearning. In recent years, a model of machine learning that detects aface direction without detecting feature points, such as eyes and anose, is proposed (reference: Fine-Grained Head Pose Estimation WithoutKeypoints (2017)). Use of such a machine learning model enablesdetection of a face direction using a face image picked up from a cameraarranged on a clavicle position.

FIG. 57 is a block diagram showing a hardware configuration of thecamera body according to this embodiment. A condenser lens 1311condenses reflected light from a user's face. A face image pickup device1312 consists of an image pickup driver, a solid state image sensor, animage signal processing circuit, etc. like the image pickup unit 40, andpicks up a face image.

In the first embodiment, the image near the user's jaw is separated fromthe background using the reflected infrared light 25. In the meantime,when a face direction is detected using the machine learning as thethirteenth embodiment, the face direction detection unit 20 does notneed the infrared LED lighting circuit. This enables to use an imagepickup unit equivalent to the image pickup unit 40 that picks up naturallight.

The face direction calculation device 1313 performs filter arithmetic asa main process of Deep Learning at high speed. The face directioncalculation device 1313 may be achieved by an exclusive processor usingASIC or FPGA, or may be achieved by the overall control CPU 101.

Parameters learned beforehand are set to the face direction calculationdevice 13143. The face direction calculation device 13143 finds angularinformation showing a face direction on the basis of the face imageoutput from the face image pickup device 1312 and the preset parameters.Learning of the parameters used for detecting the face direction needsmany learning images. Each learning image is a combination of a faceimage and information about vertical and horizontal angles of the faceas correct answers.

FIG. 58A, FIG. 58B, and FIG. 58C are schematic views showing examples oflearning images picked up under conditions of (H: 0°, V: 0°), (H: 30°,V: 0°), and (H: 0°, V: 30°), respectively. Where “H” means thehorizontal direction and “V” means the vertical direction. A pluralityof learning images are picked up while moving the face by every 10°within the face direction detection range. For example, 100 images arepicked up at each position.

For example, the face direction detection range shall be from −60° to+60° in the horizontal direction and from −60° to +50° in the verticaldirection. In such a case, learning images are picked up for every 10°in the vertical range −60° to +50° while keeping the angle in thehorizontal direction constant. These image pickup operations arerepeated for every 10° in the horizontal range −60° to +60°.

Moreover, in order to respond various users and situations, it isnecessary to cover various conditions other than the face angle inaccumulating learning images. For example, it is necessary to selecthuman objects so as to cover estimated user's physique, age, and gender,in preparing learning images. Moreover, it is necessary to preparelearning images broadly so as to absorb difference of estimatedbackgrounds, such as indoor and outdoor.

FIG. 59 is a flowchart showing a face direction detection process usingmachine learning according to the thirteenth embodiment. First, a faceimage is picked up using the image pickup unit 1311 (a step 1331). Next,the pickup face image is resized to the size suitable to input into theface direction calculation device 1314 (a step 1332). Next, the resizedface image is input into the face direction calculation device 1314, anda face direction is calculated (a step 1333).

When the process of machine learning like Deep Learning is performed,reliability showing probability of a processing result is calculated inaddition to processing results, such as a face direction, in general. Ina step 1334, it is determined whether the reliability is equal to ormore than a predetermined threshold. As a result of determination in thestep S1334, when the reliability is equal to or more than the threshold,the face direction calculated in the step S1333 is set as a new facedirection (a step 1335). That is, the face direction is updated. As aresult of determination in the step S1334, when the reliability is lessthan the threshold, the face direction is not updated.

As described above, according to the thirteenth embodiment, the facedirection is detectable using the machine learning like Deep Learning.

Next, a fourteenth embodiment will be described. The fourteenthembodiment detects a face direction using a ToF (Time of Flight) camera.FIG. 60 is a block diagram showing a hardware configuration of thecamera body according to this embodiment.

A ToF device 1411 has a light source and measures a distance to anobject using light emitted from the light source and reflected by theobject. In this embodiment, an object is a user's face.

There are two main ToF distance measurement methods. A direct ToF methodmeasures a distance on the basis of a time period from emission of alight source to reception of reflected light from an object. An indirectToF method controls a light source to emit light periodically andmeasures a distance to the object by detecting phase difference betweenemission light and reflected light. This embodiment can use any ToFmethods. The ToF device 1411 generates a distance image (depth map)showing distance information by mapping the measured distanceinformation in two dimensions.

FIG. 61A is a schematic view showing a distance image generated by theToF device 1141 arranged in a user's clavicle position and measuresupwardly. In FIG. 61A, a near part is indicated by a white area and afar part is indicated by a black area. The distance image in FIG. 61Aincludes a face area 1421 from a root of neck to a nose, and objects1422 in a background. A face-direction calculation device 1412calculates a face direction based on the distance image generated by theToF device 1411. In this embodiment, the face-direction calculationdevice 1412 shall be achieved by the overall control CPU 101. Theconfiguration is not restricted to this. For example, the face-directioncalculation device 1412 may be constituted by an exclusive CPU.

FIG. 62 is a flowchart showing a face-direction calculation process. Theoverall control CPU 101 extracts a face part from the distance imagegenerated by the ToF device 1411 (a step 1431). When the measurement isperformed by installing the ToF device 1411 in a clavicle position, theface as a measurement target is located at a short distance, and theother object will be located at a long distance. Accordingly, only aface part is extracted by applying a threshold process to the distanceimage shown in FIG. 61A. The threshold process converts a pixel of whicha pixel value is less than the threshold into a black pixel. Thethreshold may be a fixed value defined beforehand or may be calculatedaccording to contents of the distance image.

FIG. 61B is a schematic view showing an image that extracted a face partby applying the threshold process to the distance image in FIG. 61A. Asshown in FIG. 61B, since the objects 1422 in the background in FIG. 61Aare below the threshold, they are converted into black pixels, and onlythe face area 1421 is extracted.

Next, the image in FIG. 61B is divided into areas according to thedistance information (a step 1432). FIG. 61C is a schematic view showingthe image after the area division. FIG. 6C shows that the face area 1421is divided into six areas 14211 through 14216. The area 14211 is thenearest area and the area 14216 is the most distant area.

Next, a throat position (head rotation center) is extracted (a step1433).

As described in the first embodiment, the throat position is located ata center in the lateral direction in the nearest area 14211.Accordingly, a point 14217 in FIG. 61D is set as the throat position.The overall control CPU 101 extracts a chin position next (a step 1434).

As described in the first embodiment, the chin position is located inthe shorter area 14512 adjacent to the area 14214 in which the distanceincreases sharply. Accordingly, the overall control CPU 101 sets a point14218 in FIG. 61D as the chin position. The point 14218 is located in acenter of the area 14214 in the lateral direction, and is the mostdistant from the throat position 14217.

When the throat position and chin position are determined, face anglesin the horizontal and vertical directions are determined, and they arerecorded as the observation direction (direction vector) (a step S1435).The face angle in the horizontal direction is detectable on the basis ofthe throat position and chin position by the method described in thefirst embodiment. Moreover, when the distance image is obtained usingthe ToF camera, when the chin position is determined, the distance tothe chin position is fixed. Accordingly, the face angle in the verticaldirection is also detected by the method described in the firstembodiment.

The overall control CPU 101 determines the face angles in the lateraland vertical directions and saves then into the primary memory 103 as auser's observation direction vi.

As mentioned above, the face direction is detectable using a ToF cameraaccording to the fourteenth embodiment.

Although the preferable embodiments of the present disclosure aredescribed above, the present disclosure is not limited to theseembodiments. Various modifications and changes are available within thescope of the gist. Moreover, a modification that does not use a part ofthe functions is also included. Although some embodiment shows thechange of the field angle in addition to the recording direction, suchembodiments can be performed even when not changing the field angle.

Other Embodiments

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure has been described with reference toexemplary embodiments, it is to be understood that the disclosure is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Applications No.2021-040848, filed Mar. 12, 2021, and No. 2022-029152, filed Feb. 28,2022, which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. An image pickup apparatus comprising: anobservation direction detection unit that is worn on a body other than ahead of a user; an image pickup unit that is worn on the body of theuser; a memory device that stores a set of instructions; and at leastone processor that executes the set of instructions to: detect anobservation direction of the user by the observation direction detectionunit; pick up an image by the image pickup unit; and output an outputimage corresponding to the observation direction based on the imagepicked up by the image pickup unit.
 2. The image pickup apparatusaccording to claim 1, wherein the at least one processor executesinstructions in the memory device to detect the observation direction ofthe user as a three-dimensional observation direction.
 3. The imagepickup apparatus according to claim 1, wherein the at least oneprocessor executes instructions in the memory device to: output anobservation direction of the user in a lateral direction as an angle ina first detection direction, and output an observation direction of theuser in a vertical direction as an angle in a second detection directionthat is perpendicular to the first detection direction.
 4. The imagepickup apparatus according to claim 1, wherein the observation directiondetection unit comprises: an infrared irradiation unit that irradiatesan infrared irradiation surface of the user with infrared light; and aninfrared detection unit that detects reflected light of the infraredlight reflected by the infrared irradiation surface.
 5. The image pickupapparatus according to claim 4, wherein the at least one processorexecutes instructions in the memory device to: obtain distanceinformation about each of distance areas of the infrared irradiationsurface from the reflected light of the infrared light detected by theinfrared detection unit, and detect the observation direction based onthe distance information.
 6. The image pickup apparatus according toclaim 5, wherein the at least one processor executes instructions in thememory device to: detect a head rotation center and a chin position ofthe user based on the distance information, and detect the observationdirection from the head rotation center and the chin position.
 7. Theimage pickup apparatus according to claim 6, wherein the at least oneprocessor executes instructions in the memory device to: set the headrotation center at a position that is nearest to the infrared detectionunit and that is located at a center in a lateral direction in adistance area of which a relative distance to the infrared detectionunit is shortest among the distance areas.
 8. The image pickup apparatusaccording to claim 6, wherein the at least one processor executesinstructions in the memory device to: set the chin position at aposition that is farthest from the head rotation center and that is neara distance area of which a relative distance to the infrared detectionunit is relatively large among the distance areas.
 9. The image pickupapparatus according to claim 8, wherein the at least one processorexecutes instructions in the memory device to: output an observationdirection of the user in a lateral direction as an angle in a firstdetection direction and outputs an observation direction of the user ina vertical direction as an angle in a second detection direction that isperpendicular to the first detection direction, and calculate a movingangle of the chin position around the head rotation center as the anglein the first detecting direction.
 10. The image pickup apparatusaccording to claim 8, wherein the at least one processor executesinstructions in the memory device to: output an observation direction ofthe user in a lateral direction as an angle in a first detectiondirection, output an observation direction of the user in a verticaldirection as an angle in a second detection direction that isperpendicular to the first detection direction, and calculate the anglein the second detecting direction based on intensity of the reflectedlight of the chin position.
 11. The image pickup apparatus according toclaim 1, wherein the at least one processor executes instructions in thememory device to: extract an image corresponding to the observationdirection from the image picked up by the image pickup unit, and outputan extracted image.
 12. The image pickup apparatus according to claim 1,wherein the at least one processor executes instructions in the memorydevice to: perform calibration of the observation direction detectionunit using a calibrator capable of wirelessly connecting to the imagepickup apparatus, wherein the calibrator is provided with a facedetection unit that projects infrared light and detects a face of theuser, and wherein the observation direction is not detected during aperiod in which the face detection unit is projecting the infraredlight.
 13. The image pickup apparatus according to claim 1, wherein theat least one processor executes instructions in the memory device to:generate metadata including in-image position information that shows aposition and a size of an image corresponding to the observationdirection with respect to an image of each frame, and generate a videofile in which the metadata and the image of each frame are encoded forevery frame.
 14. The image pickup apparatus according to claim 13,wherein the at least one processor executes instructions in the memorydevice to: obtain an optical correction value corresponding to anoptical design of an image pickup lens in the image pickup unit, whereinthe optical correction value is included in the metadata.
 15. The imagepickup apparatus according to claim 13, wherein the at least oneprocessor executes instructions in the memory device to: detect movementof the image pickup apparatus, and obtain a moving amount, wherein themoving amount is included in the metadata.
 16. The image pickupapparatus according to claim 15, wherein the moving amount is detectedby one of an acceleration sensor that detects acceleration, an angularspeed sensor that measures an angular speed, and a magnetometric sensorthat measures a direction of a magnetic field.
 17. A portable devicecapable of wirelessly connecting to an image pickup apparatus, theportable device comprising: a memory device that stores a set ofinstructions; and at least one processor that executes the set ofinstructions to: receive a video file in which metadata, which includesin-image position information that shows a position and a size of animage corresponding to an observation direction of a user with respectto an image of each frame picked up by the image pickup apparatus, andan image of each frame are encoded for every frame; extract the metadatafrom the video file; extract the image of the frame encoded with themetadata extracted from the video file; correct the image of the frameextracted by the second extraction unit using the metadata extracted bythe first extraction unit; and record the image of the frame correctedby the frame image correction unit as a video image.
 18. The imagepickup apparatus according to claim 1, wherein the observation directiondetection unit comprises a face direction detection unit that detects aface direction of the user, wherein the at least one processor executesinstructions in the memory device to: obtain an image including apositioning index from the image pickup unit that picks up thepositioning index; detect the face direction by the face directiondetection unit during calibration; calculate a position of thepositioning index in the image picked up during the calibration from ashape of the positioning index included in the image obtained; generateinformation showing relationship between the face direction detected andthe position of the positioning index calculated; and performcalibration of a center position of a target visual field correspondingto the face direction detected by the face direction detection unitbased on the information generated.
 19. A calibrator capable ofwirelessly connecting to an image pickup apparatus, the calibratorcomprising: a face detection unit that detects a face of a user whowears the image pickup apparatus on an own body; a memory device thatstores a set of instructions; and at least one processor that executesthe set of instructions to: display a positioning index that is pickedup by an image pickup unit of the image pickup apparatus duringcalibration; and display a button that is pressed to transmit aninstruction of the calibration to the image pickup apparatus in a casewhere it is determined that the user is looking at the positioning indexaccording to a detection result of the face detection unit.
 20. Thecalibrator according to claim 19, further comprising: an angular speedsensor; and wherein the at least one processor executes instructions inthe memory device to perform calibration of a center position of atarget visual field by comparing information from the angular speedsensor with face direction information about a user that is detected bya face direction detection unit of the image pickup apparatus and iswirelessly transmitted.
 21. The calibrator according to claim 19,wherein the face detection unit picks up a face image of the user sothat the face image will include an image pickup unit that is integrallymounted on the image pickup apparatus that is worn on a body other thana head of the user together with an observation direction detection unitthat detects an observation direction of the user, and wherein the atleast one processor executes instructions in the memory device tocalculate a vertical distance between an image pickup lens of the imagepickup apparatus and an eye position of the user based on a size of theimage pickup unit in the face image.
 22. The image pickup apparatusaccording to claim 1, wherein the observation direction detection unitcomprises a face direction detection unit that detects a face directionof the user, wherein the at least one processor executes instructions inthe memory device to: calculate an angular speed of the face of the userbased on the face direction detected by the face direction detectionunit; determine a recording direction of the output image based on theobservation direction; and change the recording direction to a delayeddirection that is delayed from movement of the detected face directionin a case where it is determined, as a result of the calculation, thatthe face of the user moves at the angular speed beyond a predeterminedangular speed beyond a first predetermined period.
 23. The image pickupapparatus according to claim 22, wherein the at least one processorexecutes instructions in the memory device to output an entire imagepicked up by the image pickup unit during an image pickup operation of amoving object even if it is determined, as a result of the calculation,that the face of the user moves at the angular speed beyond thepredetermined angular speed beyond the first predetermined period. 24.The image pickup apparatus according to claim 23, wherein the at leastone processor executes instructions in the memory device to stopchanging the recording direction to the delayed direction in a casewhere a period elapsed after changing the recording direction to thedelayed direction exceeds a second predetermined period.
 25. The imagepickup apparatus according to claim 22, wherein the at least oneprocessor executes instructions in the memory device not to change therecording direction in a case where it is determined, as a result of thecalculation, that a period while the face of the user moves at theangular speed beyond a predetermined angular speed is less than thefirst predetermined period.
 26. The image pickup apparatus according toclaim 22, wherein the at least one processor executes instructions inthe memory device to add an image effect when the output image isswitched from the image of the delayed direction to the imagecorresponding to the observation direction detected by the observationdirection detection unit.
 27. The image pickup apparatus according toclaim 1, further comprising a moving-amount detection unit that detectsa moving amount of the image pickup apparatus during a video imagepickup operation by the image pickup unit, wherein the at least oneprocessor executes instructions in the memory device to: delay themoving amount of the observation direction in a case where themoving-amount detection unit detects that the image pickup apparatus isaccelerating; and accelerate the moving amount of the observationdirection to cover a delayed amount in a case where the moving-amountdetection unit detects that the image pickup apparatus is slowing down.28. The image pickup apparatus according to claim 27, wherein themoving-amount detection unit detects the moving amount by comparingimages of a plurality of frames obtained by the image pickup unit duringa video image pickup operation.
 29. The image pickup apparatus accordingto claim 27, wherein the at least one processor executes instructions inthe memory device to output a part of the image that is extractedaccording to the observation direction.
 30. The image pickup apparatusaccording to claim 27, further comprising a drive mechanism that drivean image pickup direction of the image pickup unit in a yaw directionand a pitch direction, wherein the at least one processor executesinstructions in the memory device to control the drive mechanism so asto change the image pickup direction of the image pickup unit based onthe observation direction.
 31. The image pickup apparatus according toclaim 27, wherein the at least one processor executes instructions inthe memory device to correct the moving amount of the observationdirection so that the moving speed of the observation direction willbecome approximately constant in an output video image.
 32. The imagepickup apparatus according to claim 1, wherein the observation directiondetection unit and the image pickup unit are integrally constituted,further comprising: a distance measurement unit that measures a distancefrom the image pickup unit to an image pickup target area, wherein theat least one processor executes instructions in the memory device to:create distance map information about the image pickup target area fromthe measurement result by the distance measurement unit; and calculate adirection of an observation object of the user seen from the imagepickup unit based on the observation direction, the distance mapinformation, and a vertical distance between the image pickup unit andan eye position of the user.
 33. The image pickup apparatus according toclaim 32, further comprising a posture detection unit that detects ahorizontal axis of the image pickup unit, wherein the at least oneprocessor executes instructions in the memory device to: calculate anangle formed between the horizontal axis detected and a direction of anexternal positioning index seen from the image pickup unit; calculatethe vertical distance based on the angle calculated and a distancebetween the image pickup unit and the positioning index measured by thedistance measurement unit.
 34. The image pickup apparatus according toclaim 33, wherein calibration of the observation direction detected bythe observation direction detection unit is performed based on thevertical distance calculated, an observation direction detected by theobservation direction detection unit in a case where the positioningindex is located in each designated position, and the distance betweenthe image pickup unit and the positioning index measured by the distancemeasurement unit.
 35. The image pickup apparatus according to claim 22,wherein the at least one processor executes instructions in the memorydevice to: switch an immediately preceding mode to a first image pickupmode, in which the recording direction is determined based on theobservation direction during the video image pickup operation, in a casewhere the face direction detection unit can detect the face directionduring the video image pickup operation; and switch an immediatelypreceding mode to one of other image pickup modes, in which therecording direction is determined based on a factor other than theobservation direction during the video image pickup operation, in a casewhere the face direction detection unit cannot detect the facedirection.
 36. The image pickup apparatus according to claim 35, whereinthe at least one processor executes instructions in the memory deviceto: recognize an object from an image of the recording direction of aframe of a video image picked up by the image pickup unit; set, in acase where the observation direction detection unit cannot detect theobservation direction and an identical object is recognized in a pastpredetermined period, the factor other than the observation direction toa direction that tracks the identical object; and switch an immediatelypreceding mode to a second image pickup mode that is one of the otherimage pickup modes.
 37. The image pickup apparatus according to claim36, wherein the at least one processor executes instructions in thememory device to: beforehand register an object to be detected; set, ina case where the observation direction detection unit cannot detect theobservation direction and the object registered beforehand is detectedfrom a newest pickup image, the factor other than the observationdirection to a direction that tracks the object registered beforehand;and switch an immediately preceding mode to a third image pickup modethat is one of the other image pickup modes.
 38. The image pickupapparatus according to claim 37, wherein the at least one processorexecutes instructions in the memory device to: set, in a case where theobservation direction detection unit cannot detect the observationdirection and neither an identical object nor the object registeredbeforehand can be detected, the factor other than the observationdirection to one of the observation directions detected before theobservation direction detection unit lost the observation direction andthe observation direction that is moving at a change amount before theobservation direction detection unit lost the observation direction; andswitch the immediately preceding mode to a fourth image pickup mode thatis one of the other image pickup modes.
 39. The image pickup apparatusaccording to claim 38, wherein the at least one processor executesinstructions in the memory device to widen a field angle of the image ofthe recording direction than a prescribed field angle in the fourthimage pickup mode.
 40. The image pickup apparatus according to claim 39,wherein the switching of the mode is continuously active even after themode is switched to one of the first, second, third, and fourth imagepickup modes.
 41. The image pickup apparatus according to claim 40,wherein the at least one processor executes instructions in the memorydevice to restore the widened field angle to the prescribed field anglein a case where the fourth image pickup mode is switched to one of thefirst, second, third image pickup modes.
 42. The image pickup apparatusaccording to claim 39, wherein the at least one processor executesinstructions in the memory device to: notify the user of a detectionerror of the observation direction in a case where the observationdirection detection unit cannot detect the observation direction: andnotify the user of the detection error in a case where the first imagepickup mode is switched to one of the other image pickup modes.
 43. Theimage pickup apparatus according to claim 1, wherein the observationdirection detection unit comprises a face direction detection unit thatdetects a face direction of the user, wherein the at least one processorexecutes instructions in the memory device to: detect a face directionof the user by the face direction detection unit; calculate a firstobservation direction from the face direction detected; estimate asecond observation direction from a factor other than the face directiondetected; calculate reliability of the first observation direction;determine the observation direction to the first observation directionin a case where the reliability is equal to or more than a threshold;and determine the observation direction based on the first observationdirection, the second observation direction, and the reliability in acase where the reliability is less than the threshold and the secondobservation direction is reliable.
 44. The image pickup apparatusaccording to claim 1, wherein a detection optical axis of theobservation direction detection unit and an image pickup optical axis ofthe image pickup unit are directed in mutually different directions. 45.The image pickup apparatus according to claim 44, wherein the detectionoptical axis of the observation direction detection unit is directed toa jaw of the user from the observing direction detection unit.
 46. Theimage pickup apparatus according to claim 44, wherein the image pickupoptical axis of the image pickup unit is directed to a front directionof the user from the image pickup unit.
 47. The image pickup apparatusaccording to claim 44, wherein the image pickup apparatus in which theobservation direction detection unit and the image pickup unit areintegrally constituted is built in a camera body, and wherein a lateraloverall length of the image pickup apparatus is longer than its verticaloverall length when seen from a front of the user in a state where theuser wears the camera body.
 48. The image pickup apparatus according toclaim 47, wherein the camera body is provided with fixing members thatcontact a user's body, wherein the fixing members are respectivelyarranged in vicinities of right and left ends of the image pickupapparatus in the state where the user wears the camera body.
 49. Theimage pickup apparatus according to claim 48, further comprising contactangle adjustment mechanisms that adjust angles of the fixing memberswith respect to the use's body.
 50. The image pickup apparatus accordingto claim 47, wherein the image pickup apparatus is connected with a neckhanging member for wearing the image pickup apparatus on a user's neck,wherein the neck hanging member is connected to vicinities of right andleft ends of the image pickup apparatus in the state where the userwears the camera body.
 51. The image pickup apparatus according to claim50, wherein the neck hanging member is provided with aneck-hanging-angle adjustment mechanism that adjusts an angle of theneck hanging member with respect to the image pickup apparatus.
 52. Theimage pickup apparatus according to claim 50, wherein the neck hangingmember is provided with a band part of which a sectional shape is not aperfect circle, wherein distance between right and left portions of theband part that are symmetrical to the image pickup apparatus becomesshorter toward an upper side from a lower side in the state where theuser wears the camera body.
 53. The image pickup apparatus according toclaim 50, wherein the image pickup apparatus is connected with a powersource unit through the neck hanging member, wherein the power sourceunit is arranged behind the user's neck in the state where the userwears the camera body.
 54. The image pickup apparatus according to claim53, wherein the image pickup apparatus is connected with the powersource unit through a power supply member, wherein the power supplymember is arranged inside the neck hanging member.
 55. The image pickupapparatus according to claim 22, wherein the image pickup unit has animage pickup lens and an image sensor that convers an optical imageformed by the image pickup lens into RAW data, wherein the image pickupunit outputs RAW data read from a predetermined area of the image sensoras an image picked up by the image pickup unit, wherein the at least oneprocessor executes instructions in the memory device to: extract datawithin an area, which is narrower than the predetermined area, includinga target visual field in the recording direction and a margin around thetarget visual field from the RAW data; and develop the data extracted.56. The image pickup apparatus according to claim 55, wherein the marginis a pixel area used for an image stabilization process.
 57. The imagepickup apparatus according to claim 55, wherein the at least oneprocessor executes instructions in the memory device to change a shapeof the target visual field and a shape of the margin according to therecording direction and an optical property of the image pickup lens.58. The image pickup apparatus according to claim 55, wherein the atleast one processor executes instructions in the memory device to recordthe data that is extracted and developed as an image of the recordingdirection without recording data that is not extracted from thepredetermined area.
 59. The image pickup apparatus according to claim58, wherein the at least one processor executes instructions in thememory device to transmit the image of the recording direction to anexternal appreciation device.
 60. The image pickup apparatus accordingto claim 59, wherein the external appreciation device applies an opticalcorrection process and an image stabilization process to the image ofthe recording direction, and wherein the at least one processor executesinstructions in the memory device to transmit information required forthe optical correction process and the image stabilization processtogether with the image of the recording direction.
 61. A control methodfor an image pickup apparatus, the control method comprising: detectingan observation direction of a user by an observation direction detectionunit that is worn on a body other than a head of the user; picking up animage by an image pickup unit that is worn on the body of the user; andoutputting an image corresponding to the observation direction based onthe image picked up.
 62. A control method for a portable device capableof wirelessly connecting to an image pickup apparatus, the controlmethod comprising: receiving a video file in which metadata, whichincludes in-image position information that shows a position and a sizeof an image corresponding to an observation direction of a user withrespect to an image of each frame picked up by the image pickupapparatus, and an image of each frame are encoded for every frame;extracting the metadata from the video file; extracting the image of theframe encoded with the metadata extracted from the video file;correcting the image of the frame extracted using the metadataextracted; and recording the image of the frame corrected as a videoimage.
 63. A control method for a calibrator capable of wirelesslyconnecting to an image pickup apparatus, the control method comprising:displaying a positioning index that is picked up by an image pickup unitof the image pickup apparatus during calibration; detecting a face of auser who wears the image pickup apparatus on an own body; and displayinga button that is pressed to transmit an instruction of the calibrationto the image pickup apparatus in a case where it is determined that theuser is looking at the positioning index according to a detection resultof the face detection unit.
 64. A non-transitory computer-readablestorage medium storing a control program causing a computer to execute acontrol method for an image pickup apparatus, the control methodcomprising: detecting an observation direction of a user by anobservation direction detection unit that is worn on a body other than ahead of the user; picking up an image by an image pickup unit that isworn on the body of the user; and outputting an image corresponding tothe observation direction based on the image picked up.
 65. Anon-transitory computer-readable storage medium storing a controlprogram causing a computer to execute a control method for a portabledevice capable of wirelessly connecting to an image pickup apparatus,the control method comprising: receiving a video file in which metadata,which includes in-image position information that shows a position and asize of an image corresponding to an observation direction of a userwith respect to an image of each frame picked up by the image pickupapparatus, and an image of each frame are encoded for every frame;extracting the metadata from the video file; extracting the image of theframe encoded with the metadata extracted from the video file;correcting the image of the frame extracted using the metadataextracted; and recording the image of the frame corrected as a videoimage.
 66. A non-transitory computer-readable storage medium storing acontrol program causing a computer to execute a control method for acalibrator capable of wirelessly connecting to an image pickupapparatus, the control method comprising: displaying a positioning indexthat is picked up by an image pickup unit of the image pickup apparatusduring calibration; detecting a face of a user who wears the imagepickup apparatus on an own body; and displaying a button that is pressedto transmit an instruction of the calibration to the image pickupapparatus in a case where it is determined that the user is looking atthe positioning index according to a detection result of the facedetection unit.